numpy array converted to pandas dataframe drops values

Question

I need to calculate statistics for each node of a 2D grid. I figured the easy way to do this was to take the cross join (AKA cartesian product) of two ranges. I implemented this using numpy as this function:

def node_grid(x_range, y_range, x_increment, y_increment):
    x_min = float(x_range[0])
    x_max = float(x_range[1])
    x_num = (x_max - x_min)/x_increment + 1
    y_min = float(y_range[0])
    y_max = float(y_range[1])
    y_num = (y_max - y_min)/y_increment + 1

    x = np.linspace(x_min, x_max, x_num)
    y = np.linspace(y_min, y_max, y_num)

    ng = list(product(x, y))
    ng = np.array(ng)
    return ng, x, y

However when I convert this to a pandas dataframe it drops values. For example:

In [2]: ng = node_grid(x_range=(-60, 120), y_range=(0, 40), x_increment=0.1, y_increment=0.1)
In [3]: ng[0][(ng[0][:,0] > -31) & (ng[0][:,0] < -30) & (ng[0][:,1]==10)]
Out[3]: array([[-30.9,  10. ],
   [-30.8,  10. ],
   [-30.7,  10. ],
   [-30.6,  10. ],
   [-30.5,  10. ],
   [-30.4,  10. ],
   [-30.3,  10. ],
   [-30.2,  10. ],
   [-30.1,  10. ]])

In [4]: node_df = pd.DataFrame(ng[0])
node_df.columns = ['xx','depth']
print(node_df[(node_df.depth==10) & node_df.xx.between(-30,-31)])
Out[4]:Empty DataFrame
Columns: [xx, depth]
Index: []

The dataframe isn't empty:

In [5]: print(node_df.head())
Out[5]:      xx  depth
0 -60.0    0.0
1 -60.0    0.1
2 -60.0    0.2
3 -60.0    0.3
4 -60.0    0.4

values from the numpy array are being dropped when they are being put into the pandas array. Why?

Could you please specify the product function you are using. As posted the code does not work for me. — Ascurion
– Ascurion, Commented Aug 24, 2016 at 21:34

scottD · Accepted Answer · 2016-08-24 22:17:35Z

0

the "between" function demands that the first argument be less than the latter.

In: print(node_df[(node_df.depth==10) & node_df.xx.between(-31,-30)]) xx depth 116390 -31.0 10.0 116791 -30.9 10.0 117192 -30.8 10.0 117593 -30.7 10.0 117994 -30.6 10.0 118395 -30.5 10.0 118796 -30.4 10.0 119197 -30.3 10.0 119598 -30.2 10.0 119999 -30.1 10.0 120400 -30.0 10.0

For clarity the product() function used comes from the itertools package, i.e., from itertools import product

edited Aug 24, 2016 at 22:17

answered Aug 24, 2016 at 21:50

scottD

162 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ascurion · Accepted Answer · 2016-08-24 21:49:13Z

0

I can't fully reproduce your code.

But I find the problem is that you have to turn the lower and upper boundaries around in the between query. The following works for me:

print(node_df[(node_df.depth==10) & node_df.xx.between(-31,-30)])

when using:

ng = np.array([[-30.9,  10. ],
                [-30.8,  10. ],
                [-30.7,  10. ],
                [-30.6,  10. ],
                [-30.5,  10. ],
                [-30.4,  10. ],
                [-30.3,  10. ],
                [-30.2,  10. ],
                [-30.1,  10. ]])
node_df = pd.DataFrame(ng)

answered Aug 24, 2016 at 21:49

Ascurion

5233 silver badges13 bronze badges

Collectives™ on Stack Overflow

numpy array converted to pandas dataframe drops values

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related