How to process Unicode CSV files in Ruby CSV correctly?

Question

I am trying to process some Google Adwords csv files. The files are available in UNICODE format. When I use Ruby CSV parser to parse the file. I am not able to read the file. The characters display as \x00a \x00b etc.

I ended up having to open the file in OpenOffice and choose UTF-8 to render the file and then save it. After that, Ruby CSV can process the file. I also have to remove the first character in the csv file that looks like number 8 in black circle because it is not a valid UTF-8 character. This special character was the result of UNICODE to UTF-8 conversion in OpenOffice.

So what is the best way to convert the csv file to a Ruby friendly encoding without illegal characters?

To see what I can mean, you can try open Ruby CSV to open this file and parse the lines.

https://github.com/zben/encoding_test/blob/master/encoding_test.csv

file says encoding_test.csv: Little-endian UTF-16 Unicode text — icktoofay
– icktoofay, Commented Apr 13, 2013 at 22:36

icktoofay · Accepted Answer · 2013-04-13 22:38:36Z

0

This page suggests using Iconv.iconv to convert:

doc = Iconv.iconv('UTF-8', 'UTF-16', doc)

answered Apr 13, 2013 at 22:38

icktoofay

130k23 gold badges261 silver badges239 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to process Unicode CSV files in Ruby CSV correctly?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related