Given a numpy array, I want to identify which rows contain NaN values and objects. For example, a row would contain both a float value and a list.
For the input array arr, I tried doing arr[~np.isnan(arr).any(axis=1)] but then I get the error message
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could
not be safely coerced to any supported types according to the casting rule ''safe''
In [314]: x = np.array([[1, [2,3], np.nan], [3, [5,6,7], 8]])
In [315]: x
Out[315]:
array([[1, list([2, 3]), nan],
[3, list([5, 6, 7]), 8]], dtype=object)
In [316]: x.shape
Out[316]: (2, 3)
In [317]: x[0]
Out[317]: array([1, list([2, 3]), nan], dtype=object)
In [318]: x[1]
Out[318]: array([3, list([5, 6, 7]), 8], dtype=object)
isnan works on a float dtype array; object dtype can't be converted to that:
In [320]: np.isnan(x)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-320-3b2be83a8ed7> in <module>
----> 1 np.isnan(x)
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
We can, though, test the elements one by one with a is np.nan test:
In [325]: np.frompyfunc(lambda i: i is np.nan,1,1)(x)
Out[325]:
array([[False, False, True],
[False, False, False]], dtype=object)
frompyfunc returns an object dtype; let's convert that to bool:
In [328]: np.frompyfunc(lambda i: i is np.nan,1,1)(x).astype(bool)
Out[328]:
array([[False, False, True],
[False, False, False]])
In [329]: np.any(_, axis=1) # test whole rows
Out[329]: array([ True, False])
In [330]: x[~_, :] # use that as mask to keep other rows
Out[330]: array([[3, list([5, 6, 7]), 8]], dtype=object)
The pandas isnull suggested in the other answer, can do a similar element by element test:
In [335]: pd.isnull(x)
Out[335]:
array([[False, False, True],
[False, False, False]])
arr[~np.isnan(arr, casting = 'no').any(axis=1)]orarr[~np.isnan(arr, casting = 'unsafe').any(axis=1)]np.isnanis intended for nummeric, floating point type arrays - so maybe you can pre-select that part of your array which can be converted to float bevor applyingnp.isnan?np.array([5., np.nan, 'asdf'])- gives me a similar / the sameTypeErroras @Anonymous experiences.object.np.array([[1, [2,3], np.nan], [3, [5,6,7], 8]]), I would want to get rid of the first row.