Using UDF in a DataFrame

Question

I have a dataframe dfmmmIncOther:

 dfmmmIncOther=dfmmmIncOther.agg(max("time_res"),min("time_res"),avg("time_res")).withColumn("typestat",lit("IQ_SU5"))
        .withColumnRenamed("max(time_res)","delay max")
        .withColumnRenamed("min(time_res)","delay min")
        .withColumnRenamed("avg(time_res)","delay moy")

Type of time_res is a minutes

I did a function to convert minutes to hours, then I convert it on UDF to use it after:

// convert hours on udf

 val convertHours : (Int) => String =(input: Int) => {
      val minutes = input%60
      val hours   = input/60
      "%sh:%sm".format(hours,minutes)
    }

    val udfconvertHours = udf(convertHours)*

I changed the variable dfmmmIncOther, to convert minutes to hours:

dfmmmIncOther=dfmmmIncOther.withColumn("delaymax",udfconvertHours(col("delay max"))).withColumn("delaymin",udfconvertHours(col("delay min"))).withColumn("delaymoy",udfconvertHours(col("delay moy")))

the spark interpreter is return a big exception, I think my fault in syntax but don't know where's exactly.

Some remark from you, I will be appreciate

can you share the exception as well? can you share the schema or data of dfmmnIncOther — koiralo
– koiralo, Commented Jul 19, 2018 at 12:03

koiralo · Accepted Answer · 2018-07-19 12:17:23Z

One of the main that you are doing wrong reassigning the variable dfmmmIncOther

You can use new variable to store it rather than to reassign it

Here is the simple example

import spark.implicits._

//sample data 
val dfmmmIncOther = Seq(120, 122, 12, 68, 123, 435, 234).toDF("time_res")

//create an UDF 
val udfconvertHours = udf((input: Int) => "%sh:%sm".format(input/60,input%60))

//calculate and apply udf
var result = dfmmmIncOther.agg(
  max("time_res").as("max"),
  min("time_res").as("min"),
  avg("time_res").as("avg")
  )
  .withColumn("typestat", lit("IQ_SU5"))
  .withColumn("delaymax",udfconvertHours(col("max")))
  .withColumn("delaymin",udfconvertHours(col("min")))
  .withColumn("delaymoy",udfconvertHours(col("avg")))

result.show(false)

Output:

+---+---+------------------+--------+--------+--------+--------+
|max|min|avg               |typestat|delaymax|delaymin|delaymoy|
+---+---+------------------+--------+--------+--------+--------+
|435|12 |159.14285714285714|IQ_SU5  |7h:15m  |0h:12m  |2h:39m  |
+---+---+------------------+--------+--------+--------+--------+

Collectives™ on Stack Overflow

Using UDF in a DataFrame

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related