How To Convert An Elixir Binary To A String?

Question

So I'm trying to convert a binary to a string. This code:

t = [{<<71,0,69,0,84,0>>}]
String.from_char_list(t)

But I'm getting this when I try this conversion:

** (ArgumentError) argument error
    (stdlib) :unicode.characters_to_binary([{<<70, 0, 73, 0, 78, 0>>}])
    (elixir) lib/string.ex:1161: String.from_char_list/1

I'm assuming the <<70, 0, etc. is likely a list of graphemes (it's the return from an API call and the API is not quite documented) but do I need to specify the encoding somehow?

I know I'm likely missing something obvious (maybe that's not the right function to use?) but I can't seem to figure out what to do here.

EDIT:

For what it's worth, the binary above is the return value of an Erlang ODBC call. After a little more digging I found that the binary in question is actually a "Unicode binary encoded as UTF16 little endian" (see here: http://www.erlang.org/doc/apps/odbc/odbc.pdf pg. 9 re: SQL_WVARCHAR) Doesn't really change the issue but it does add some context.

Elixir assumes strings are UTF8 encoded binaries, not UTF16. — rvirding
– rvirding, Commented Mar 21, 2014 at 14:14

sebisnow · Accepted Answer · 2019-07-18 08:57:11Z

28

There's a couple of things here:

1.) You have a list with a tuple containing one element, a binary. You can probably just extract the binary and have your string. Passing the current data structure to to_string is not going to work.

2.) The binary you used in your example contains 0, an unprintable character. In the shell, this will not be printed properly as a string, due to the fact that Elixir can't tell the difference between just a binary, and a binary representing a string, when the binary representing a string contains unprintable characters.

3.) You can use pattern matching to convert a binary to a particular type. For instance:

iex> raw = <<71,32,69,32,84,32>>
...> Enum.join(for <<c::utf8 <- raw>>, do: <<c::utf8>>)
"G E T "
...> <<c::utf8, _::binary>> = raw
"G"

Also, if you are getting binary data from a network connection, you probably want to use :erlang.iolist_to_binary, since the data will be an iolist, not a charlist. The difference is that iolists can contain binaries, nested lists, as well as just be a list of integers. Charlists are always just a flat list of integers. If you call to_string, on an iolist, it will fail.

edited Jul 18, 2019 at 8:57

sebisnow

2,15722 silver badges26 bronze badges

answered Mar 19, 2014 at 22:27

bitwalker

9,2891 gold badge39 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Onorio Catenacci Over a year ago

I thought that the list containing tuple was an issue but I wanted to give the code exactly as it comes back from the API. I'm guessing the result is in a DBCS. Need to dig into it a bit further.

Onorio Catenacci Over a year ago

Partially I was hoping there might be something already built into the library that I was missing. Good to know I didn't miss anything.

bitwalker Over a year ago

Yes it does seem like each character is being stored in two bytes, good call! The distinction between binaries and strings and charlists and iolists is confusing at times, but I think there are some changes coming down the pipe that should make it more obvious when to use each one.

Onorio Catenacci Over a year ago

Yep, it's actually a little endian UTF16 binary. See my edit to my question.

Kshitij Mittal Over a year ago

When I tried <<t::utf8>> = <<71,32,69,32,84,32>>, in elixir console, I got ** (MatchError) no match of right hand side value: "G E T " .. Can anyone explain why?

|

Gus · Accepted Answer · 2018-03-15 17:22:45Z

7

I made a function to convert binary to string

def raw_binary_to_string(raw) do
   codepoints = String.codepoints(raw)  
      val = Enum.reduce(codepoints, 
                        fn(w, result) ->  
                            cond do 
                                String.valid?(w) -> 
                                    result <> w 
                                true ->
                                    << parsed :: 8>> = w 
                                    result <>   << parsed :: utf8 >>
                            end
                        end)

  end

Executed on iex console

iex(6)>raw=<<65, 241, 111, 32, 100, 101, 32, 70, 97, 99, 116, 117, 114, 97, 99, 105, 111, 110, 32, 65, 99, 116, 117, 97, 108>>
iex(6)>raw_binary_to_string(raw)
iex(6)>"Año de Facturacion Actual"

edited Mar 15, 2018 at 17:22

Gus

901 silver badge9 bronze badges

answered Mar 11, 2017 at 7:01

Andres Garcia

811 silver badge3 bronze badges

Comments

sebisnow · Accepted Answer · 2019-07-18 09:17:14Z

5

Not sure if OP has since solved his problem, but in relation to his remark about his binary being utf16-le: for specifically that encoding, I found that the quickest (and to those more experienced with Elixir, probably-hacky) way was to use Enum.reduce:

# coercing it into utf8 gives us ["D", <<0>>, "e", <<0>>, "v", <<0>>, "a", <<0>>, "s", <<0>>, "t", <<0>>, "a", <<0>>, "t", <<0>>, "o", <<0>>, "r", <<0>>]
<<68, 0, 101, 0, 118, 0, 97, 0, 115, 0, 116, 0, 97, 0, 116, 0, 111, 0, 114, 0>>  
|> String.codepoints()
|> Enum.reduce("", fn(codepoint, result) ->
                     << parsed :: 8>> = codepoint
                     if parsed == 0, do: result, else: result <> <<parsed>>
                   end)

# "Devastator"
|> IO.puts()

Assumptions:

utf16-le encoding
the codepoints are backwards-compatible with utf8 i.e. they use only 1 byte

Since I'm still learning Elixir, it took me a while to get to this solution. I looked into other libraries people made, even using something like iconv at a bash level.

edited Jul 18, 2019 at 9:17

sebisnow

2,15722 silver badges26 bronze badges

answered Sep 20, 2016 at 18:25

user701847

3374 silver badges15 bronze badges

1 Comment

Cheezmeister Over a year ago

"backwards-compatible with utf8" is a misleading. "Representable with ASCII" might be more accurate. UTF-8 is backwards-compatible with ASCII (UTF-16 is not), but it also has 2-, 3-, or 4-byte characters. UTF-16 always uses 2 bytes, which is often wasteful. Anyway, I'm an Elixir newb and yeah this is a hack.

Jbur43 · Accepted Answer · 2021-10-11 19:42:33Z

4

Ecto.UUID.load/1 will convert a binary to string and return a tuple:

binary = Ecto.UUID.bingenerate()
<<99, 148, 189, 126, 144, 154, 71, 236, 160, 110, 149, 143, 67, 162, 177, 192>>

Ecto.UUID.load(binary)
{:ok, "6394bd7e-909a-47ec-a06e-958f43a2b1c0"}

credit: https://stackoverflow.com/a/43530427/2091331

answered Oct 11, 2021 at 19:42

Jbur43

1,31219 silver badges38 bronze badges

1 Comment

Peter R Over a year ago

convert a binary to string and return a tuple. No it won't. It will convert a hex encoded uuid into a UUID string, but it will not convert any given binary to a string. Even if it did, it also inserts hyphens. It's only to be used for UUIDs.

rvirding · Accepted Answer · 2014-03-21 14:13:33Z

3

The last point definitely does change the issue, and explains it. Elixir uses binaries as strings but assumes and demands that they are UTF8 encoded, not UTF16.

answered Mar 21, 2014 at 14:13

rvirding

21k2 gold badges39 silver badges57 bronze badges

Comments

Pikender Sharma · Accepted Answer · 2018-10-26 18:10:49Z

2

In reference to http://erlang.org/pipermail/erlang-questions/2010-December/054885.html

You can use :unicode.characters_to_list(binary_string, {:utf16, :little}) to verify result and store too

IEX eval

iex(1)> y                                                
<<115, 0, 121, 0, 115, 0>>
iex(2)> :unicode.characters_to_list(y, {:utf16, :little})
'sys'

Note : Value printed as sys for <<115, 0, 121, 0, 115, 0>>

answered Oct 26, 2018 at 18:10

Pikender Sharma

312 bronze badges

Comments

Thomas Thornier · Accepted Answer · 2020-01-20 00:16:46Z

1

You can use Comprehensions

    defmodule TestModule do
      def convert(binary) do
        for c <- binary, into: "", do: <<c>>
      end
    end
    TestModule.convert([71,32,69,32,84,32]) |> IO.puts

answered Jan 20, 2020 at 0:16

Thomas Thornier

34 bronze badges

1 Comment

Onorio Catenacci Over a year ago

If I pass the original argument <<71,0,69,0,84,0>> in this case, this doesn't work. I get an argument error. If I have to take the additional step of converting the binary to a list, you probably want to specify that in your answer as well.

Collectives™ on Stack Overflow

How To Convert An Elixir Binary To A String?

7 Answers 7

6 Comments

Comments

1 Comment

1 Comment

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

6 Comments

Comments

1 Comment

1 Comment

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related