How to create a new column in pyspark?

Question

In my pyspark DataFrame I have two columns price1 and price2. I want to create a new column result based on the formula ((price1 - price2)/price1). However, I want also to check that neither price1 nor price2 are null, and price1 is not 0.

How can I correctly create a new column using these conditions?

Now I have this:

df = df.withColumn("result", df["price1"]-df["price2"]/df["price1"])

what result do you expect if price1==0 or if one prices is null? — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Oct 25, 2017 at 11:07
@MaxU: Sorry, I have not specified it. If price1==0 or any price is equal to null, then I expect result to be equal to 0. — Dinosaurius
– Dinosaurius, Commented Oct 25, 2017 at 11:09

MaxU - stand with Ukraine · Accepted Answer · 2017-10-25 11:43:15Z

1

I think you can do it this way:

df = df.withColumn("result", df["price1"]-df["price1"]/df["price2"]).fillna(0)

answered Oct 25, 2017 at 11:43

MaxU - stand with Ukraine

212k37 gold badges402 silver badges436 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Suresh · Accepted Answer · 2017-10-25 11:40:19Z

0

If you can use udf,

from pyspark.sql import functions as F

udf = F.udf(lambda x,y : 0 if x == 0 or not all((x,y)) else x-y/x)
df = df.withColumn("result", udf(df["price1"],df["price2"]))

answered Oct 25, 2017 at 11:40

Suresh

5,8802 gold badges27 silver badges42 bronze badges

Comments

undefined_variable · Accepted Answer · 2017-10-25 11:42:23Z

0

df = df.withColumn("result", 
when(df.col("price1").isNull OR df.col("price2").isNull OR df.col("price1")==0,0)
.otherwise(df.col("price1")-df.col("price2")/df.col("price1")))

This is how it can be done using scala..

answered Oct 25, 2017 at 11:42

undefined_variable

6,2282 gold badges24 silver badges38 bronze badges

Collectives™ on Stack Overflow

How to create a new column in pyspark?

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related