How to insert NaN array into a numpy 2D array

Question

I'm trying to insert an arbitrary number of rows of NaN values within a 2D array at specific places. I'm logging some data from a microcontroller in a .csv file and parsing with python.

The data is stored in a 3 column 2D array like this

[(122.0, 1.0, -47.0) (123.0, 1.0, -47.0) (125.0, 1.0, -44.0) ..., 
(39.0, 1.0, -47.0) (40.0, 1.0, -45.0) (41.0, 1.0, -47.0)]

The first column is an sequence counter. What I'm trying to do is iterate through the sequence values, diff current and previous sequence number and insert as many rows with nan as there are missing sequences.

Basically,

[(122.0, 1.0, -47.0) (123.0, 1.0, -47.0) (125.0, 1.0, -44.0)]

would become

[(122.0, 1.0, -47.0) (123.0, 1.0, -47.0) (nan, nan, nan) (125.0, 1.0, -44.0)]

However the following implementation of np.insert produces an error

while (i < len(list[1])):
     pid = list[i][0]
     newMissing = (pid - LastGoodId + 255) % 256
     TotalMissing = TotalMissing + newMissing
     np.insert(list,i,np.zeros(newMissing,1) + np.nan)   
     i = i + newMissing
     list[i][0] = TotalMissing
     LastGoodId = pid

---> 28 np.insert(list,i,np.zeros(newMissing,1) + np.nan) 29 i = i + newMissing 30 list[i][0] = TotalMissing

TypeError: data type not understood

Any ideas on how I can accomplish this?

It is always a good idea to include the error message you get when you say something is not working. — P. Camilleri
– P. Camilleri, Commented Oct 4, 2016 at 9:32
Is the first col sorted? What happens when the first col goes from 125.0 to 39.0, should we consider those as missing elements in between them? By missing elements, do you always mean in ascending order? — Divakar
– Divakar, Commented Oct 4, 2016 at 10:23
Yes, the data in the first column is incremental and wraps around 8 bit overflow(0 - 255) so if the first col goes form 125 to 39 it means 168 packets were lost. — Hungi Mahar
– Hungi Mahar, Commented Oct 4, 2016 at 10:47
That's an important piece of info. Please add that into the question. — Divakar
– Divakar, Commented Oct 4, 2016 at 11:14

P. Camilleri · Accepted Answer · 2016-10-04 09:29:19Z

4

From the doc of np.insert():

import numpy as np
a = np.arrray([(122.0, 1.0, -47.0), (123.0, 1.0, -47.0), (125.0, 1.0, -44.0)]))
np.insert(a, 2, np.nan, axis=0)
array([[ 122.,    1.,  -47.],
       [ 123.,    1.,  -47.],
       [  nan,   nan,   nan],
       [ 125.,    1.,  -44.]])

answered Oct 4, 2016 at 9:29

P. Camilleri

13.3k10 gold badges49 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Hungi Mahar Over a year ago

Tried np.insert(a, 2, np.nan, axis=0) and got the following error message: TypeError: expected a readable buffer object

Divakar · Accepted Answer · 2016-10-04 11:29:47Z

Approach #1

We can use an initialization based approach to handle multiple gaps and gaps of any lengths -

# Pre-processing step to create monotonically increasing array for first col
id_arr = np.zeros(arr.shape[0])
id_arr[np.flatnonzero(np.diff(arr[:,0])<0)+1] = 256
a0 = id_arr.cumsum() + arr[:,0]

range_arr = np.arange(a0[0],a0[-1]+1)
out = np.full((range_arr.shape[0],arr.shape[1]),np.nan)
out[np.in1d(range_arr,a0)] = arr

Sample run -

In [233]: arr      # Input array
Out[233]: 
array([[ 122.,    1.,  -47.],
       [ 123.,    1.,  -47.],
       [ 126.,    1.,  -44.],
       [  39.,    1.,  -47.],
       [  40.,    1.,  -45.],
       [  41.,    1.,  -47.]])

In [234]: out
Out[234]: 
array([[ 122.,    1.,  -47.],
       [ 123.,    1.,  -47.],
       [  nan,   nan,   nan],
       [  nan,   nan,   nan],
       [ 126.,    1.,  -44.],
       [  nan,   nan,   nan], (168 NaN rows)
       .....
       [  nan,   nan,   nan],
       [  nan,   nan,   nan],
       [  39.,    1.,  -47.],
       [  40.,    1.,  -45.],
       [  41.,    1.,  -47.]])

Approach #2

An alternative approach could be suggested to handle such generic cases using np.insert instead of initialization, like so -

idx = np.flatnonzero(~np.in1d(range_arr,a0))
out = np.insert(arr,idx - np.arange(idx.size),np.nan,axis=0)

Collectives™ on Stack Overflow

How to insert NaN array into a numpy 2D array

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related