Combining multiple columns into one column in pyspark dataframe

Question

I need to merge multiple columns of a data frame into one single column as below in pyspark. I am new to pyspark.

Dataframe input and output

These columns were actually key value pairs(10,20,30) at the source and I tried to explode but I am getting 'Column' object is not callable. The existing logic splits the column as mentioned above in the input data frame and hence wanted to proceed with the same.

Can you show the python code you tried? That makes it easier to see where the issue is. — Neele22
– Neele22, Commented Oct 25, 2023 at 14:40

Neele22 · Accepted Answer · 2023-10-25 14:47:07Z

0

It is difficult to know where the issue is with what you already tried without seeing your code, but you could use PySpark's coalesce:

from pyspark.sql.functions import coalesce, col

df.withColumn("NEW", coalesce(col("MARK1"), col("MARK2"), col("MARK3")))

And subsequently drop the MARK1, MARK2, and MARK3 columns and then rename NEW to MARK1.

answered Oct 25, 2023 at 14:47

Neele22

4725 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Combining multiple columns into one column in pyspark dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related