CSV Returning as Bytes not String Python

Question

The error I am recieving is this:

_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

Where in my code would I open the file in text mode?

import csv
import urllib.request
import pymysql.cursors
connection = pymysql.connect(host='localhost',
                         user='',
                         password='',
                         db='mydb',
                         charset='utf8',
                         cursorclass=pymysql.cursors.DictCursor)
try:
    url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?
    collectionName=Health%20Care&format=csv'
    response =  urllib.request.urlopen(url)
    csv_data = csv.reader(response)
    for row in csv_data:

    cursor.execute('INSERT INTO Financials (names, \
          classes, mark )' \
          'VALUES("%s", "%s", "%s")', 
          row)
finally:
    connection.close()

you could slightly fix your indentation as well

Jean-François Fabre
– Jean-François Fabre ♦

2018-10-20 09:32:37 +00:00
Commented Oct 20, 2018 at 9:32 — Jean-François Fabre
– Jean-François Fabre ♦, Commented Oct 20, 2018 at 9:32

Jean-François Fabre · Accepted Answer · 2018-10-20 09:40:23Z

2

response =  urllib.request.urlopen(url)

here response returns an iterator on a bytes object,.

If you know that the csv file is just plain text, you could just insert a generator comprehension to decode the lines:

csv_data = csv.reader(line.decode() for line in response)

or map (don't you love python 3 map when you can use it without lambda?)

csv_data = csv.reader(map(bytes.decode,response))

self-contained example:

import urllib.request,csv

url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?collectionName=Health%20Care&format=csv'
response =  urllib.request.urlopen(url)
for row in csv.reader(line.decode() for line in response):
    print(row)

now you're feeding csv.reader with a list of strings, which will work.

sample output:

['CAPR', 'Capricor Therapeutics Inc.', 'NASDAQ Capital Market', 'Healthcare', 'close', '0.9011', '1539955800519', '0.875', '1539979200341', '0.9011', '0.8702', '0.875', 'Close', 'October 19, 2018', '1539979200341', '96625', '', '', '', '0.875', '1539979200341', '0.9011', '0.0261', '0.02983', '1539954158310', '0.9011', '-0.0261', '-0.02896', '', '', '106532', '', '', '', '', '26905263', '-1.72', '3.19', '0.851', '-0.4716351592356688']
['AVDL', 'Avadel Pharmaceuticals plc', 'NASDAQ Global Market', 'Healthcare', 'close', '4.24', '1539955800643', '4.1', '1539979200312', '4.24', '4', '4.1', 'Close', 'October 19, 2018', '1539979200312', '78386', '', '', '', '4.1', '1539979200312', '4.2', '0.1', '0.02439', '1539892803066', '4.2', '-0.1', '-0.02381', '', '', '114387', '', '', '', '', '150734712', '-3.8', '11.93', '3.98', '-0.5579009090909092']

edited Oct 20, 2018 at 9:40

answered Oct 20, 2018 at 9:30

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Aran-Fey Over a year ago

Note that this will blindly decode the response content with the utf-8 codec, which might not match the actual encoding. Most servers will include the correct encoding in the HTTP headers. There's a question about finding the correct response encoding here, but none of the answers there are really good.

Jean-François Fabre Over a year ago

@Aran-Fey correct. In that case that works (tested with the real site). This encoding guess game is tough.

Collectives™ on Stack Overflow

CSV Returning as Bytes not String Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related