Adding rows to a numpy array without losing previous data stored

Question

I am reading data from a file and extracting the information that I want from it, as floats. I am then storing this data into a temporary list and using vstack to try and put the data into an array with each row being new data that is being processed.

for line in lines:
    if line.find('GPS')!=-1:
        funcGPS(line)
    if line.find('-----')!=-1:
        MasterArray = numpy.vstack(temp)
        temp = []
        #print MasterArray
    if line.find('SERVO')!=-1:
        funcSERVO(line)

This is how I am trying to copy the data to the array. I am successfully extracting the data and after extracting the data obtained I want to add the data to the array. Currently, it is copying the data over the previous data when I build onto MasterArray. Is there anyway to do this without specifically stating what the size of MasterArray is? I don't want to limit the amount of data that can be obtained.

Thanks!

Where does temp come from? You can use np.append instead of np.vstack inside your loop, or else you can keep adding it to a list and vstack that list after your loop, but you might be better off loading all your data in at once, with np.genfromtxt, and then editing it. — askewchan
– askewchan, Commented Sep 10, 2013 at 16:25
We probably can't help unless we know what temp is and how it's constructed. vstack accepts a sequence of arrays, and joins them together. So temp should be a tuple or list of arrays to join; if you mean to append to MasterArray, then your definition of temp will need to look something like temp = (MasterArray, newrow). This isn't the best approach to dynamic array creation though; see here for more information. — senderle
– senderle, Commented Sep 10, 2013 at 16:38
temp is a 1xn matrix ( currently 1x20 ). Append works but it adds it to the end of the column and I want it to be a new row. temp comes from this line of code which is extracting the data from the file. for element in listline: value = element.split(':') value=value[1].split('}') float(value[0]) #print value[0] temp.append(float(value[0])) This is done twice before being placed into the MasterArray. Thank you for the help! — Rubedo
– Rubedo, Commented Sep 10, 2013 at 19:05

Bi Rico · Accepted Answer · 2013-09-10 16:42:25Z

2

Growing an array in a loop is considered bad practice. If you know the final size, or a reasonable upper limit, of the array it's best to pre-allocate the array and then fill it. If you don't know the final size you can often use a temporary list and convert it to an array after the loop. This is not unique to numpy, you'll see similar usage patterns in matlab and c programs.

Do something like this if you know the final size of the array or a reasonable upper limit.

N = 100
array = np.zeros(100)
count = 0
for line in file:
    array[count] = ...
    count += 1
 array = array[:count] # Or maybe array[:count].copy()

or something like this if you don't.

temp = []
for line in file:
    temp.append(...)
array = np.array(temp)

edited Sep 10, 2013 at 16:42

answered Sep 10, 2013 at 16:37

Bi Rico

25.9k3 gold badges57 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rubedo Over a year ago

Thanks! I don't know the size, but its going to be extremely large. I am appending the data to temp and then I wanted to stack it into another array so all the data is in one matrix.

Collectives™ on Stack Overflow

Adding rows to a numpy array without losing previous data stored

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related