How to drop empty rows from a DataFrame when 'pd.notnull' does not work? Python

Question

I have a DataFrame with two columns 'A' and 'B'. My goal is to delete rows where 'B' is empty. Others have recommended to use df[pd.notnull(df['B'])]. For example here: Python: How to drop a row whose particular column is empty/NaN?

However, somehow this does not work in this case. Why not and how to solve this?

    A          B
0   Lorema     Ipsuma
1   Corpusa    Dominusa
2   Loremb     
3   Corpusc    Dominusc
4   Loremd     
5   Corpuse    Dominuse

This is the desired result:

    A          B
0   Lorema     Ipsuma
1   Corpusa    Dominusa
2   Corpusc    Dominusc
3   Corpuse    Dominuse

You have to understand what values are in your data frames. For example, if it is an empty string, df[df.B != ""] would do. If it is None or NA, then notnull() shoud lwork etc — rafaelc
– rafaelc, Commented Jul 4, 2018 at 21:12
Hint: try print(type(df['B'].iloc[2])) with your dataframe above to see what type you have. — jpp
– jpp, Commented Jul 4, 2018 at 21:14
So you should try @Wen's solution.. looks like you have empty strings. — jpp
– jpp, Commented Jul 4, 2018 at 21:16

rafaelc · Accepted Answer · 2018-07-04 21:29:45Z

0

Basically, you could have whitespaces, tabs or even a \n in these blank cells.

For all those cases, you can strip values first, and then remove the rows, i.e.

df[df.B.str.strip().ne("") & df.B.notnull()]

I believe this should cover all cases.

answered Jul 4, 2018 at 21:29

rafaelc

59.4k15 gold badges64 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

twhale Over a year ago

The empty cells are cells that have never been filled after the column was created. This solution does not yet do the trick. Would it be possible to fill those cells with something (e.g. '0') and then drop?

rafaelc Over a year ago

Can you post how you are creating this data frame? If a csv file, how you post a sample csv?

twhale Over a year ago

My function is complicated. But it either returns a string to the cell or it returns a white space ''. (So I expected your solution to work.) And then writes the result to a csv file.

rafaelc Over a year ago

Following jpps' suggestion, what does df['B'].iloc[2] == '' print ?

twhale Over a year ago

It prints False.

|

Collectives™ on Stack Overflow

How to drop empty rows from a DataFrame when 'pd.notnull' does not work? Python

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related