Remove duplicate in an array[string]

Question

I have a dataframe column with array of string as below. (Key,value) pair

ColA
[(1,2),(1,3),(1,4),(2,3)]

I have to remove duplicate keys by min value and get the results. Dont want to explode and do it. Key should be unique and the key is picked based on the min value. In the above column, there are three pairs with key as 1. So should pick (1,2) since value 2 is min among (1,2),(1,3),(1,4)

Output should be: ColA [(1,2),(2,3)]

I created a udf like

Val removeDup = udf((arr: Seq[String]) => {
Arr.map(x=>x.split(","))}))

Cannot use reduceby key as its a dataframe/dataset.

@Deek can you provide sample input data for the dataframe you have? — Anahcolus
– Anahcolus, Commented Jul 25, 2017 at 5:21

Glennie Helles Sindholt · Accepted Answer · 2017-07-25 07:48:21Z

1

Okay, so provided that the column if of type String and not type Seq[String], the code below should give you what you want:

val removeDup = udf((str: String) => {
  str.split("\\(|\\)").filter(s => s != "," && s != "").map(s => {
    val array = s.replace("(", "").replace(")", "").split(",")
    (array(0), array(1))
  })
  .groupBy(_._1)
  .mapValues(a => a.sortBy(_._2).head)
  .values
  .toSeq
  .sortBy(_._1)
})

On your example:

val df = spark.sparkContext.parallelize(Seq("(1,2),(1,3),(1,4),(2,3)").toDF("colA")

df.select(removeDup('colA)).show

this yields

+--------------------+
|           UDF(colA)|
+--------------------+
|      [[1,2], [2,3]]|
+--------------------+

If you wish to keep the column type as String, you would need to add .mkString(",") to the udf.

answered Jul 25, 2017 at 7:48

Glennie Helles Sindholt

13.2k6 gold badges47 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ɢʀᴜɴᴛ · Accepted Answer · 2017-07-25 06:13:10Z

0

df.select("v1","v2").groupby("v2").min("v2").show

edited Jul 25, 2017 at 6:13

ɢʀᴜɴᴛ

33k15 gold badges123 silver badges115 bronze badges

answered Jul 25, 2017 at 5:53

Robin

6951 gold badge7 silver badges10 bronze badges

1 Comment

user0712 Over a year ago

Its a single column with array for string. This might not help!

Collectives™ on Stack Overflow

Remove duplicate in an array[string]

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related