How to load data with numpy with no fixed column size

Question

How can we load a text file with tab delimited values but with no fixed column size in the way that the missing values are skipped completely ending up with a list/array or whatever container containing numpy arrays for each line (or a whole numpy array? -> might be impossible, because numpy needs fixed sizes)?

Is this only possible by reading in each line with python and then converting with loadtxt the line into a 1D array?

list=[]
for lineString in file:
    list.append( np.loadtxt(lineString) )

or is it possible somehow with load txt?

Lee · Accepted Answer · 2014-03-05 11:52:46Z

1

Maybe you could use pandas

If your file looks like this:

1   2   3   4   5   6
1   2
8.0 9   97  54

Then doing this:

import pandas as pd
pd.read_csv('yourfile.txt',sep='\t')

gives:

   1  2   3   4   5   6
0  1  2 NaN NaN NaN NaN
1  8  9  97  54 NaN NaN

To convert to a numpy array:

np.array(pd.read_csv('yourfile.txt',sep='\t'))


array([[  1.,   2.,  nan,  nan,  nan,  nan],
       [  8.,   9.,  97.,  54.,  nan,  nan]])

answered Mar 5, 2014 at 11:52

Lee

31.4k31 gold badges124 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to load data with numpy with no fixed column size

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related