Issue adding new columns to dataframe using pyspark

Question

Say I run this

DF1.withColumn("Is_elite",
               array_intersect(DF1.year,DF1.elite_years))
    .show()

I get the result I want which is a new column called Is_elite with the correct values and all Then in the next command I run

DF1.show

It just shows me what DF1 would have looked like had I not run the first command, my column is missing.

H Roy · Accepted Answer · 2020-04-20 04:04:26Z

2

Since you have added .show() method in the line, it is not returning a new data frame. Make the following changes and try it out

elite_df = DF1.withColumn("Is_elite",array_intersect(DF1.year,DF1.elite_years))
elite_df.show()

In case you get confused about the object in python, try to print the type of object.

#the following must return a dataframe object. 
print(type(elite_df))

Dataframes are immutable and every transformation create a new dataframe reference and hence if you try to print the old datagram, you will not get the revised result.

answered Apr 20, 2020 at 4:04

H Roy

6356 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Issue adding new columns to dataframe using pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related