Reading non-uniform data from file into array with NumPy

Question

Suppose I have a text file that looks like this:

33 3
46 12
23 10 23 11 23 12 23 13 23 14 23 15 23 16 24 10 24 11 24 12 24 13 24 14 24 15 24 16 25 14 25 15 25 16 26 16 27 16 28 16 29 16
33 17 33 18 33 19 34 17 34 18 34 19 35 17 35 18 35 19 36 19
41 32 41 33 42 32 42 33

I would like to read each line into a separate array of integers, as in (pseudo code):

for line in textfile:  
    currentArray = firstLine  
    do stuff with currentArray

where in the first iteration, currentArray would be

array([33, 3])

and in the second iteration, currentArray would be

array([46, 12])

until the last iteration, when currentArray would be

array([41, 32, 41, 33, 42, 32, 42, 33])

Basically, I would like to have the functionality of the numpy function loadtxt:

currentArray = loadtxt('scienceVertices.txt', usecols=() )

Except instead of usecols, being able to specify the row, e.g.,

currentArray = loadtxt('scienceVertices.txt', userows=(line) )

Depending on the size of your data file, it may be significantly more efficient not to load all the lines into memory at once. This is possible, since iterating over a file object in Python will give you the lines one at a time, loaded sequentially from disc with IO buffering. OTOH, if you only have a few megabytes of data it may be easier not to bother. — Katriel
– Katriel, Commented Feb 11, 2011 at 21:59
@katrielalex In my application I will likely only be reading kilobytes of data, but I'll keep your suggestion in mind if I move onto larger files. Thanks. — lookitsmarc
– lookitsmarc, Commented Feb 11, 2011 at 22:59

Paul · Accepted Answer · 2011-02-11 21:48:07Z

17

Here's a one-liner:

arrays = [np.array(map(int, line.split())) for line in open('scienceVertices.txt')]

arrays is a list of numpy arrays.

answered Feb 11, 2011 at 21:48

Paul

43.9k17 gold badges112 silver badges126 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

lookitsmarc Over a year ago

Well played. I didn't ask for it to be in a list of numpy arrays but I think this is better for my application. Thank you.

heracho Over a year ago

This solution, at least in python3, return a list of maps. How can I use something similar to retrieve numpy arrays?

Paul Over a year ago

@heracho Just wrap the map(...) function with a list() function.

payne · Accepted Answer · 2011-02-11 21:47:30Z

6

for line in textfile:
  a = np.array([int(v) for v in line.strip().split(" ")])
  # Work on your array

edited Feb 11, 2011 at 21:47

answered Feb 11, 2011 at 21:35

payne

14.3k5 gold badges44 silver badges49 bronze badges

Comments

Sven Marnach · Accepted Answer · 2011-02-13 17:14:21Z

3

You can also use numpy.fromstring()

for line in f:
    a = numpy.fromstring(line.strip(), dtype=int, sep=" ")

or -- if you want full flexibility -- even numpy.loadtxt():

for line in f:
    a = numpy.loadtxt(StringIO.StringIO(line), dtype=int)

For long lines, these solution will perform better than the Python code in the other answers.

answered Feb 13, 2011 at 17:14

Sven Marnach

608k123 gold badges966 silver badges865 bronze badges

Comments

Santa · Accepted Answer · 2011-02-11 21:52:39Z

2

f = open("file", "r")
array = []
line = f.readline()
index = 0
while line:
    line = line.strip("\n")
    line = line.split()
    array.append([])
    for item in line:
        array[index].append(int(item))
    line = f.readline()
    index += 1
f.close()

print array

edited Feb 11, 2011 at 21:52

Santa

11.6k8 gold badges54 silver badges64 bronze badges

answered Feb 11, 2011 at 21:43

Asterisk

3,5742 gold badges37 silver badges55 bronze badges

Collectives™ on Stack Overflow

Reading non-uniform data from file into array with NumPy

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related