Pyspark Dataframe - How to concatenate columns based on array of columns as input

Question

I have dataframe of 10 columns and want to do function - concatenation based on Array of columns which come as input:

arr = ["col1", "col2", "col3"]

This is current so far:

newDF = rawDF.select(concat(col("col1"), col("col2"), col("col3") )).exceptAll(updateDF.select( concat(col("col1"), col("col2"), col("col3") ) ) )

Also:

df3 = df2.join(df1, concat( df2.col1, df2.col2, df2.col3, df2.col3 ) == df1.col5 )

But I want to make a loop or function to do this based on input array (not hard-coding it as is now).

What is the best way?

can you post your expected output?

Jay Kakadiya
– Jay Kakadiya

2020-02-21 19:29:46 +00:00
Commented Feb 21, 2020 at 19:29 — Jay Kakadiya
– Jay Kakadiya, Commented Feb 21, 2020 at 19:29

murtihash · Accepted Answer · 2020-02-21 21:32:16Z

1

You can unpack the cols using (*). In the pyspark.sql docs, if any functions have (*cols), it means that you can unpack the cols. For concat:

pyspark.sql.functions.concat(*cols)

from pyspark.sql import functions as F
arr = ["col1", "col2", "col3"]
newDF = rawDF.select(F.concat(*(F.col(col) for col in arr))).exceptAll(updateDF.select(F.concat(*(F.col(col) for col in arr))))

For joins:

arr=['col1','col2','col3']
df3 = df2.join(df1, F.concat(*(F.col(col) for col in arr)) == df1.col5 )

edited Feb 21, 2020 at 21:32

answered Feb 21, 2020 at 18:52

murtihash

8,4401 gold badge16 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

con Over a year ago

verbal explanations are often helpful

Joe Over a year ago

Also - how would do this part? df3 = df2.join(df1, concat( df2.col1, df2.col2, df2.col3, df2.col3 ) == df1.col5 )

Collectives™ on Stack Overflow

Pyspark Dataframe - How to concatenate columns based on array of columns as input

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related