Loading whole csv columns into an array?

Question

import csv 
import numpy as np
import matplotlib.pyplot as plt

myfile = open('LoggedData_CalInertialAndMag.csv', 'rt')
reader = csv.reader(myfile)
next(reader)

a = [row[4] for row in reader]
b = [row[5] for row in reader]

So this is the beginning of my code that I was trying to load the whole column from the csv file to an array, and cast them to float later for other uses. However, I got errors so I checked len(a) and len(b) seperately. While len(a) has the length 838, which len(b) should have the same, but it's 0.

Why? and then I changed my code using append, which I feel is a bit more complex. So where might I be wrong?

It will be a very sharp arrow : )

fedorqui
– fedorqui

2015-04-10 14:00:33 +00:00
Commented Apr 10, 2015 at 14:00 — fedorqui
– fedorqui, Commented Apr 10, 2015 at 14:00
I think you mean "array" not "arrow".

ralfe
– ralfe

2015-04-10 14:01:31 +00:00
Commented Apr 10, 2015 at 14:01 — ralfe
– ralfe, Commented Apr 10, 2015 at 14:01

martineau · Accepted Answer · 2017-01-02 19:03:10Z

1

The problem is your first list comprehension for list a "consumes" the csv file as it iterates over it, so it's empty for the one for list b. Although not as concise as list comprehensions, I would suggest you create both lists at the same time. Also note the proper way to open a csv file for the csv module depends on the version of Python you're using.

import collections
import csv
import sys

csv_read_args = ({'mode': 'rb'} if sys.version_info[0] < 3 else
                 {'mode': 'r', 'newline': ''})

with open('LoggedData_CalInertialAndMag.csv', **csv_read_args) as myfile:
    reader = csv.reader(myfile)
    next(reader)
    a, b = [], []
    # feed generator expression into a zero-length deque to consume it
    generator = ((a.append(row[4]), b.append(row[5])) for row in reader)
    collections.deque(generator, maxlen=0)

An alternative to doing it this way would be to do a myfile.seek(0) between the two list comprehension statements to "rewind" the file back to the beginning. This would be less efficient because it requires the reader to parse the entire file twice.

Update

Here's another, slightly (8%-10%) faster, alternative:

with open('LoggedData_CalInertialAndMag.csv', **csv_read_args) as myfile:
    reader = csv.reader(myfile)
    next(reader)
    a, b = map(list, zip(*[(row[4], row[5]) for row in reader]))

You may not need the final map(list, ...) depending on whether you require a and b to be lists or not (zip returns a tuple in Python 2 and an iterator in Python 3).

edited Jan 2, 2017 at 19:03

answered Apr 10, 2015 at 14:25

martineau

124k29 gold badges181 silver badges319 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3431800 Over a year ago

I have used an alternative which uses append as I said in the post. I didn't use collection but your way is basically the same as what I am doing now.

martineau Over a year ago

I used collections.deque in an effort to provide something that was both efficient and concise that would allow appending data from each row to more than one list at time. I got the idea from an itertools module recipe called consume. Since deque is provided in a built-in library, it's very fast.

John · Accepted Answer · 2015-04-10 16:41:18Z

1

This should get your code working

import csv

import matplotlib.pyplot as plt
import numpy as np

a, b = [], []
with open("LoggedData_CalInertialAndMag.csv", 'rb') as f:
    reader = csv.reader(f)
    for row in reader:
        a.append(float(row[4]))
        b.append(float(row[5]))

since you appear to be using numpy and mention reading it into arrays I assume something like numpy.genfromtxt should also work

import numpy as np

filename = "LoggedData_CalInertialAndMag.csv"
a = np.genfromtxt(filename, usecols=[4], delimiter=',', dtype=float)
b = np.genfromtxt(filename, usecols=[5], delimiter=',', dtype=float)

answered Apr 10, 2015 at 16:41

John

13.8k7 gold badges53 silver badges104 bronze badges

Comments

John · Accepted Answer · 2015-04-10 16:30:44Z

0

Try adding a call to myfile.seek(0) before your second call https://docs.python.org/2/library/csv.html

read is reading the content of the file, when you try to load b all the file was already loaded, so there's nothing to look in the file :)

But a better approach would be to load both at once and then you can assign them to different arrays without reading the file again (disk I/O operations are quite expensive):

a = [[row[4], row[5]] for row in reader]
b = [r[0] for r in a]
c = [r[1] for r in a]

edited Apr 10, 2015 at 16:30

John

13.8k7 gold badges53 silver badges104 bronze badges

answered Apr 10, 2015 at 14:03

dfranca

5,3622 gold badges36 silver badges65 bronze badges

4 Comments

user3431800 Over a year ago

By seek, you meant run this function after I 've loaded the first time?

dfranca Over a year ago

Yes, but a better approach would be to load both at once, I'll edit with this alternative.

user3431800 Over a year ago

i actually need them in two different arrays

dfranca Over a year ago

So, just assign them to 2 different arrays, I'll edit the answer.

Collectives™ on Stack Overflow

Loading whole csv columns into an array?

3 Answers 3

2 Comments

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related