Python: How to read line by line in a numpy array?

Question

I would like to know can we read line by line in an array. For example:

array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

To read the first line as in array form to perform some operation and then continue with second row array:

array([0.28,0.22,0.23,0.27])

What produces the above array is because of these two lines of code:

from numpy import genfromtxt
single=genfromtxt('single.csv',delimiter=',')

single.csv

0.28,  0.22,  0.23,  0.27
0.12,  0.29,  0.34,  0.21
0.44,  0.56,  0.51,  0.65

Using readlines() seem like producing list instead of array. In my case, I'm using csv file. I'm trying to use the rows of values line by line instead of using them all together to avoid memory error. Can anyone help me?

with open('single.csv') as single:
    single=single.readlines()

Does your csv have array([0.28,0.22,0.23,0.27]) in it? That isn't csv format. — Open AI - Opting Out
– Open AI - Opting Out, Commented Feb 23, 2016 at 13:16
readlines produces a list of strings. Parse each line to get the numbers — hpaulj
– hpaulj, Commented Feb 23, 2016 at 13:58

Community · Accepted Answer · 2017-05-23 12:02:14Z

10

you can use np.fromstring

import numpy as np
with open('single.csv') as f:
    lines=f.readlines()
    for line in lines:
        myarray = np.fromstring(line, dtype=float, sep=',')
        print(myarray)

see http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.fromstring.html and How to read csv into record array in numpy?

edited May 23, 2017 at 12:02

CommunityBot

11 silver badge

answered Feb 23, 2016 at 13:37

maazza

7,25116 gold badges67 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Xiong89 Over a year ago

Sorry I'm not familiar with np.fromstring. And it returned this error: NameError: name 'double' is not defined

hpaulj · Accepted Answer · 2016-02-23 17:17:18Z

Seems you don't have experience reading files in Python. Let me work through an example in some detail, in an Ipython iteractive session

Create a multiline text to simulate your file

In [23]: txt="""0.28,  0.22,  0.23,  0.27
0.12,  0.29,  0.34,  0.21
0.44,  0.56,  0.51,  0.65"""

split it into lines to simulate the result of readlines

In [24]: txt=txt.splitlines(True)

In [25]: txt
Out[25]: 
['0.28,  0.22,  0.23,  0.27\n',
 '0.12,  0.29,  0.34,  0.21\n',
 '0.44,  0.56,  0.51,  0.65']

I can transform it into an array with genfromtxt (you could have passed the result to readlines to genfromtxt like this.

In [26]: np.genfromtxt(txt, delimiter=',')
Out[26]: 
array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

I can iterate over the lines, strip off the \n and split on ','

In [27]: for line in txt:
    print line.strip().split(',')
   ....:     
['0.28', '  0.22', '  0.23', '  0.27']
['0.12', '  0.29', '  0.34', '  0.21']
['0.44', '  0.56', '  0.51', '  0.65']

I can transform each string into a float with a list comprehension:

In [28]: for line in txt:                                  
    print [float(x) for x in line.strip().split(',')]
   ....:     
[0.28, 0.22, 0.23, 0.27]
[0.12, 0.29, 0.34, 0.21]
[0.44, 0.56, 0.51, 0.65]

Or by putting the iteration in another list comprehension, I can get a list of lists of numbers:

In [29]: data=[[float(x) for x in line.strip().split(',')] for line in  txt]

In [30]: data
Out[30]: [[0.28, 0.22, 0.23, 0.27], [0.12, 0.29, 0.34, 0.21], [0.44, 0.56, 0.51, 0.65]]

I can turn that into an array

In [31]: np.array(data)
Out[31]: 
array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

genfromtxt is essentially going through that sequence - reading the lines,spliting them, converting strings to values, and finally creating an array from the list.

There are short cuts, but I think you would benefit from working through these steps in detail. It's as much an exercise in basic Python string and list manipulation as it is about arrays.

Fauxpas · Accepted Answer · 2016-02-23 13:14:28Z

0

for list in array:
    print(list)
    for item in list:
        print(item)

gives the output:

[0.28, 0.22, 0.23, 0.27]
0.28
0.22
0.23
0.27
[0.12, 0.29, 0.34, 0.21]
0.12
0.29
0.34
0.21
[0.44, 0.56, 0.51, 0.65]
0.44
0.56
0.51
0.65

answered Feb 23, 2016 at 13:14

Fauxpas

866 bronze badges

Collectives™ on Stack Overflow

Python: How to read line by line in a numpy array?

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related