Pandas DataFrame Isnull Multiple Columns at Once

Question

Given the following:

import pandas as pd
import numpy as np

df = pd.DataFrame({'a':[np.nan,1,2],'b':[np.nan,np.nan,4]})

     a    b
0  NaN  NaN
1  1.0  NaN
2  2.0  4.0

How do I return rows where both columns 'a' and 'b' are null without having to use pd.isnull for each column?

Desired result:

     a    b
0  NaN  NaN

I know this works (but it's not how I want to do it):

df.loc[(pd.isnull(df['a']) & (pd.isnull(df['b'])]

I tried this:

df.loc[pd.isnull(df[['a', 'b']])]

...but got the following error:

ValueError: Cannot index with multidimensional key

Thanks in advance!

Quang Hoang · Accepted Answer · 2021-03-26 17:45:01Z

7

You are close:

df[~pd.isnull(df[['a', 'b']]).all(1)]

Or

df[df[['a','b']].isna().all(1)]

How about:

df.dropna(subset=['a','b'], how='all')

edited Mar 26, 2021 at 17:45

answered Mar 26, 2021 at 17:42

Quang Hoang

151k11 gold badges64 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Quang Hoang Over a year ago

@DanceParty2 was just about to update. Done!

Dance Party2 Over a year ago

Perfect! Thanks so much. Also, what does .all(1) do?

Quang Hoang Over a year ago

that's shorthand for all(axis=1), i.e. checking row-wise, default is axis=0, checking column-wise.

RavinderSingh13 · Accepted Answer · 2021-03-26 17:55:08Z

4

With your shown samples, please try following. Using isnull function here.

mask1 = df['a'].isnull()
mask2 = df['b'].isnull()
df[mask1 & mask2]

Above answer is with creating 2 variables for better understanding. In case you want to use conditions inside df itself and don't want to create condition variables(mask1 and mask2 in this case) then try following.

df[df['a'].isnull() & df['b'].isnull()]

Output will be as follows.

    a   b
0   NaN NaN

edited Mar 26, 2021 at 17:55

answered Mar 26, 2021 at 17:49

RavinderSingh13

135k14 gold badges61 silver badges100 bronze badges

Comments

ThePyGuy · Accepted Answer · 2021-03-26 18:06:01Z

2

You can use dropna() with parameter as how=all

df.dropna(how='all')

Output:

   a    b
1  1.0  NaN
2  2.0  4.0

Since the question was updated, you can then create masking either using df.isnull() or using df.isna() and filter accordingly.

df[df.isna().all(axis=1)]
   a   b
0 NaN NaN

edited Mar 26, 2021 at 18:06

answered Mar 26, 2021 at 17:45

ThePyGuy

18.5k5 gold badges24 silver badges55 bronze badges

1 Comment

ThePyGuy Over a year ago

The question was updated.. I'll update the answer accordingly

Collectives™ on Stack Overflow

Pandas DataFrame Isnull Multiple Columns at Once

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related