Pyspark: Filtering rows on multiple columns

Question

I have a dataframe and I want to filter data as below condition.

df = spark.createDataFrame([(101, 1, 2, 12, 34), (102, 1, 1, 45, 12), (103, 1, 34, 10, 0), (104, None, 2, 91, 76), (
    105, 2, 67, 23, 54), (107, 34, 255, 234, 99), (109, None, 1, 23, 55)], ['ID', 'A', 'B', 'C', 'D'])
df.show()

+---+----+---+---+---+
| ID|   A|  B|  C|  D|
+---+----+---+---+---+
|101|   1|  2| 12| 34|
|102|   1|  1| 45| 12|
|103|   1| 34| 10|  0|
|104|null|  2| 91| 76|
|105|   2| 67| 23| 54|
|107|  34|255|234| 99|
|109|null|  1| 23| 55|
+---+----+---+---+---+

filtering condition

If column A = 1, select row
If column A is null then check the column B and if column B = 1, select row

Expected output

+---+----+---+---+---+
| ID|   A|  B|  C|  D|
+---+----+---+---+---+
|101|   1|  2| 12| 34|
|102|   1|  1| 45| 12|
|103|   1| 34| 10|  0|
|109|null|  1| 23| 55|
+---+----+---+---+---+

My code:

import pyspark.sql.functions as F
df = df.filter((F.col('A') ==1 )& (F.col('A').isNull()) & (F.col('B') ==1))
df.show()

I am getting empty dataframe. Can someone help me. Thanks.

Marioanzas · Accepted Answer · 2021-03-25 08:47:46Z

3

You need | condition instead of & between condition 1 and 2:

import pyspark.sql.functions as F
df.filter((F.col('A') == 1 ) | (F.col('A').isNull() & (F.col('B') ==1)))

edited Mar 25, 2021 at 8:47

Marioanzas

1,9452 gold badges21 silver badges38 bronze badges

answered Mar 25, 2021 at 5:50

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark: Filtering rows on multiple columns

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related