Pandas - Create multiple filters and apply to dataframe

Question

I have created multiple filters for a Dataframe:

filt1 = ~df["message"].str.contains("<Media omitted>", na=False),
filt2 = ~df["message"].str.contains("http://", na=False),
filt3 = ~df["message"].str.contains("Dropped pin", na=False),

I can filter the dataframe using:

df[filt1 & filt2 & filt3]

But as I add more filters this seems like a stupid way to filter. How do I apply multiple filters to a dataframe?

I tried adding each filter to a list doing df[filterlist] and df[*filterlist] but these do not work.

jezrael · Accepted Answer · 2020-08-24 11:36:50Z

3

You can use np.logical_and.reduce:

filterlist = [filt1, filt2, filt3]

df[np.logical_and.reduce(filterlist)]

Or concat with DataFrame.all for test all Trues per rows:

df[pd.concat(filterlist, axis=1).all(axis=1)]

If possible use | for regex or:

filt = ~df["message"].str.contains("<Media omitted>|http://|Dropped pin", na=False)

edited Aug 24, 2020 at 11:36

answered Aug 24, 2020 at 11:25

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Dishin Goyani · Accepted Answer · 2020-08-24 11:34:17Z

0

You can use join to merge list for regex

df[~df.message.str.contains('|'.join(filterlist))]

answered Aug 24, 2020 at 11:34

Dishin Goyani

7,7533 gold badges33 silver badges42 bronze badges

Collectives™ on Stack Overflow

Pandas - Create multiple filters and apply to dataframe

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related