Pyspark replace string from column based on pattern from another column

Question

I have a dataframe with a text column and a name column. I would like to check if the name exists in the text column and if it does to replace it with some value. I was hoping that the following would work:

df = df.withColumn("new_text",regex_replace(col("text),col("name"),"NAME"))

but Column is not iterable so it does not work. Do I have to write a udf to do that? How would that look like?

Possible duplicate of stackoverflow.com/questions/45615621/… — giser_yugang
– giser_yugang, Commented May 30, 2019 at 9:21
Possible duplicate of Using a column value as a parameter to a spark DataFrame function — pault
– pault, Commented May 30, 2019 at 14:05
@giser_yugang this is pyspark your linked question is about scala. — Idodo
– Idodo, Commented Dec 7, 2020 at 10:19

Shantanu Sharma · Accepted Answer · 2019-05-30 09:58:00Z

7

You are almost close. Here is detailed example with withColumn and selectExpr options:

Sample df

df = spark.createDataFrame([('This is','This'),
('That is','That'),
('That is','There')],
['text','name'])

#+-------+-----+
#|   text| name|
#+-------+-----+
#|This is| This|
#|That is| That|
#|That is|There|
#+-------+-----+

Option 1: withColumn using expr function

from pyspark.sql.functions import expr, regexp_replace

df.withColumn("new_col1",expr("regexp_replace(text,name,'NAME')")).show()

#+-------+-----+--------+
#|   text| name|new_col1|
#+-------+-----+--------+
#|This is| This| NAME is|
#|That is| That| NAME is|
#|That is|There| That is|
#+-------+-----+--------+

Option 2: selectExpr using regexp_replace

 from pyspark.sql.functions import regexp_replace


df.selectExpr("*",
          "regexp_replace(text,name,'NAME') AS new_text").show()

#+-------+-----+--------+
#|   text| name|new_text|
#+-------+-----+--------+
#|This is| This| NAME is|
#|That is| That| NAME is|
#|That is|There| That is|
#+-------+-----+--------+

edited May 30, 2019 at 9:58

answered May 30, 2019 at 9:42

Shantanu Sharma

4,1491 gold badge22 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user43395 Over a year ago

Do you happen to know how to handle the case when name is a regex expression? I'm seeing an issue with expr("regexp_replace(column, 'regex', 'replace_value')")

user43395 Over a year ago

To add, it's because the 'regex' is regex, but it being surrounded as a string from expr seems to interfere.

user43395 Over a year ago

I think I solved it, but not sure why. ^([^.]+)?\\. worked instead of ^.*?\\. (but the latter works when I don't use expr)

Collectives™ on Stack Overflow

Pyspark replace string from column based on pattern from another column

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related