How to load multiple columns with numpy loadtext?

Question

I am trying to load about 10 columns of data into numpy loadtxt. I want each column to be a separate numpy array. How can I do this? Only two columns are actually being imported, I need all 10. Here's my code:

import sys
import numpy as np 
import scipy.stats

data = np.loadtxt('AD_hw6.txt')

p = data[:,0] #pressure in hpa
z = data[:,1] #height in m
t = data[:,2] #tempertature in degrees celcius
dp = data[:,3] #dewpoint in degrees celcius
rh = data[:,4] #relative humidity (%)
mr = data[:,5] #mixing ratio in g/kg

This is only 5 but I'm using 10.

This should work fine, so your data file is probably incompatible. As per Coldspeed's comment, we'd need to see some (part of the) data file to see what the actual problem is. — user707650
– user707650, Commented Oct 8, 2017 at 23:11
Before trying to allocate the data to many variables, make sure you can load the the data as one array. What is the shape and dtype of data? You may need to fiddle with delimiter, skip-header and other parameters. — hpaulj
– hpaulj, Commented Oct 8, 2017 at 23:22
There are some very detailed answers below, but I think what the OP really needs is an introduction to pandas. — Victor Chubukov
– Victor Chubukov, Commented Oct 9, 2017 at 0:31

hpaulj · Accepted Answer · 2017-10-09 00:45:39Z

With a copy-n-paste from the WY link I create a file with

-----------------------------------------------------------------------------
   PRES   HGHT   TEMP   DWPT   RELH   MIXR   DRCT   SKNT   THTA   THTE   THTV
    hPa     m      C      C      %    g/kg    deg   knot     K      K      K
-----------------------------------------------------------------------------
 1000.0    116
  971.0    357    8.0    6.0     87   6.07    200      5  283.5  300.6  284.6
  956.0    487    7.2    5.0     86   5.75    215     18  284.0  300.2  285.0
  942.1    610    8.0    5.5     84   6.06    230     30  286.0  303.2  287.1
  933.0    691    8.6    5.9     83   6.28    233     30  287.4  305.3  288.5
  925.0    763    8.0    4.9     81   5.90    235     30  287.5  304.4  288.5
  908.2    914    7.2    3.2     76   5.32    245     29  288.2  303.5  289.1
 ...

I can load it as a 2d array with:

In [1]: data = np.genfromtxt('stack46636938.txt',skip_header=5)
In [2]: data.shape
Out[2]: (35, 11)
In [3]: data[:3]
Out[3]: 
array([[ 971.  ,  357.  ,    8.  ,    6.  ,   87.  ,    6.07,  200.  ,
           5.  ,  283.5 ,  300.6 ,  284.6 ],
       [ 956.  ,  487.  ,    7.2 ,    5.  ,   86.  ,    5.75,  215.  ,
          18.  ,  284.  ,  300.2 ,  285.  ],
       [ 942.1 ,  610.  ,    8.  ,    5.5 ,   84.  ,    6.06,  230.  ,
          30.  ,  286.  ,  303.2 ,  287.1 ]])

Notice that I skipped the line with only two values.

Now I can do

p = data[:,0]

I could also use the unpack option

In [7]: p,z,t = np.genfromtxt('stack46636938.txt',skip_header=5,unpack=True,usecols=range(3))

This is the equivalent of using Python unpacking with the transpose of the 2d array:

In [9]: p,z,t = data[:,:3].T

I could also load the data as a structured array with:

In [11]: names = '   PRES   HGHT   TEMP   DWPT   RELH   MIXR   DRCT   SKNT   THT
    ...: A   THTE   THTV'.split()
In [12]: names
Out[12]: 
['PRES',
 'HGHT',
 'TEMP',
 'DWPT',
 ....
 'THTV']
In [13]: data = np.genfromtxt('stack46636938.txt', skip_header=5, dtype=None, names=names)
In [14]: data[:3]
Out[14]: 
array([( 971. , 357,  8. ,  6. , 87,  6.07, 200,  5,  283.5,  300.6,  284.6),
       ( 956. , 487,  7.2,  5. , 86,  5.75, 215, 18,  284. ,  300.2,  285. ),
       ( 942.1, 610,  8. ,  5.5, 84,  6.06, 230, 30,  286. ,  303.2,  287.1)],
      dtype=[('PRES', '<f8'), ('HGHT', '<i4'), ('TEMP', '<f8'), ('DWPT', '<f8'), ('RELH', '<i4'), ('MIXR', '<f8'), ('DRCT', '<i4'), ('SKNT', '<i4'), ('THTA', '<f8'), ('THTE', '<f8'), ('THTV', '<f8')])

and access fields by name

In [15]: data['PRES']
Out[15]: 
array([ 971. ,  956. ,  942.1,  933. ,  925. ,  908.2,  900. ,  875. ,
        850. ,  842.8,  827. ,  811.4,  792. ,  786. ,  781. ,  776. ,
        752.1,  736. ,  726. ,  724.1,  721. ,  714. ,  710. ,  706. ,
        703. ,  701. ,  700. ,  690. ,  671. ,  670.8,  669. ,  666. ,
        662. ,  645.4,  639. ])

loadtxt also works with the right row skip and unpack:

np.loadtxt('stack46636938.txt',skiprows=5,unpack=True)

If the short row is included I get an error. It starts expecting two columns and then objects when it gets 11.

GoldenGenova · Accepted Answer · 2017-10-08 23:26:06Z

0

I think the problem is that the first column row only has 2 values; I suggest you fill in zeros or NaN for the empty rows in the columns.

answered Oct 8, 2017 at 23:26

GoldenGenova

541 silver badge6 bronze badges

Comments

GoldenGenova · Accepted Answer · 2018-01-21 19:17:27Z

0

I think the problem is that the first column row only has 2 values; I suggest you fill in zeros or NaN for the empty rows in the columns.

edited Jan 21, 2018 at 19:17

answered Oct 8, 2017 at 23:08

GoldenGenova

541 silver badge6 bronze badges

3 Comments

astrogirl79 Over a year ago

I copied and pasted all of the data from this website into a text file.

astrogirl79 Over a year ago

Sorry here is the link weather.uwyo.edu/cgi-bin/…

astrogirl79 Over a year ago

I need to each column of the data to be a numpy array

Collectives™ on Stack Overflow

How to load multiple columns with numpy loadtext?

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related