Ruby/Rails CSV parsing, invalid byte sequence in UTF-8

Question

I am trying to parse a CSV file generated from an Excel spreadsheet.

Here is my code

require 'csv'
file = File.open("input_file")
csv = CSV.parse(file)

But I get this error

ArgumentError: invalid byte sequence in UTF-8

I think the error is because Excel encodes the file into ISO 8859-1 (Latin-1) and not in UTF-8

Can someone help me with a workaround for this issue, please

Thanks in advance.

In case you need to support various encodings and detect at import, Charlock Holmes worked great for me. See stackoverflow.com/a/12234195/1343535 — dbenton
– dbenton, Commented Feb 5, 2018 at 17:35

Linuxios · Accepted Answer · 2015-02-19 00:57:43Z

74

You need to tell Ruby that the file is in ISO-8859-1. Change your file open line to this:

file=File.open("input_file", "r:ISO-8859-1")

The second argument tells Ruby to open read only with the encoding ISO-8859-1.

edited Feb 19, 2015 at 0:57

answered Dec 5, 2011 at 1:38

Linuxios

35.9k13 gold badges96 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jacklin Over a year ago

This was giving me fits too, and your solution is working so far for me! Thanks!

jbnunn Over a year ago

worked like a champ. I was doing an iconv -f ISO-8859-1 -t utf-8 oldfilename > newfilename before i found this answer.

Linuxios Over a year ago

@jnunn: So glad I could help! Ruby encodings are harry things, and not that easy to deal with.

unmultimedio Over a year ago

if you're here and using 'roo' gem, here the docs: github.com/roo-rb/roo#csv-support says you can send it as a symbol: s = Roo::CSV.new("mycsv.csv", csv_options: {encoding: Encoding::ISO_8859_1})

Sebastián Palma · Accepted Answer · 2018-04-04 20:58:01Z

36

Specify the encoding with encoding option:

CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
  ...
end

edited Apr 4, 2018 at 20:58

Sebastián Palma

33.6k6 gold badges45 silver badges65 bronze badges

answered Jul 1, 2016 at 7:53

Sudhir Vishwakarma

80511 silver badges15 bronze badges

1 Comment

Abram Over a year ago

I prefer declaring keyword arguments explicitly +1 for encoding:

adamkonrad · Accepted Answer · 2013-05-22 06:09:58Z

11

You can supply source encoding straight in the file mode parameter:

CSV.foreach( "file.csv", "r:windows-1250" ) do |row|
   <your code>
end

answered May 22, 2013 at 6:09

adamkonrad

7,1222 gold badges38 silver badges42 bronze badges

1 Comment

ltrainpr Over a year ago

This worked in Ruby 2.1.5 but you have to do encoding: 'iso-8859-1' instead of "r:windows-1250".

ToTenMilan · Accepted Answer · 2019-01-22 17:07:12Z

1

If you have only one (or few) file, so when its not needed to automatically declare encoding on whatever file you get from input, and you have the contents of this file visible in plaintext (txt, csv etc) separated with i.e. semicolon, you can create new file with .csv extension manually, and paste the contents of your file there, then parse the contents like usual.

Keep in mind, that this is a workaround, but in need of parsing in linux only one big excel file, converted to some flavour of csv, it spares time on experimenting with all those fancy encodings

answered Jan 22, 2019 at 17:07

ToTenMilan

6201 gold badge10 silver badges22 bronze badges

Comments

Eliza A · Accepted Answer · 2014-05-13 06:38:19Z

0

Save the file in utf-8, unless for some reason you need to save it differently in which case you may specify the encoded set while reading the file

answered May 13, 2014 at 6:38

Eliza A

1525 bronze badges

Comments

Gagan Gami · Accepted Answer · 2014-05-13 07:14:08Z

0

add second argument "r:ISO-8859-1" as File.open("input_file","r:ISO-8859-1" )

answered May 13, 2014 at 7:14

Gagan Gami

10.3k1 gold badge32 silver badges56 bronze badges

Comments

user3787971 · Accepted Answer · 2015-02-04 16:12:37Z

0

I had this same problem and was just using google spreadsheets and then downloading as a CSV. That was the easiest solution.

Then I came across this gem

https://github.com/singlebrook/utf8-cleaner

Now I don't need to worry about this issue at all. Hope this helps!

answered Feb 4, 2015 at 16:12

user3787971

4574 silver badges22 bronze badges

Collectives™ on Stack Overflow

Ruby/Rails CSV parsing, invalid byte sequence in UTF-8

7 Answers 7

4 Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

4 Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Linked

Related