use spark SQL udf in dataframe API

Question

How can I use a UDF which works great in spark like

sparkSession.sql("select * from chicago where st_contains(st_makeBBOX(0.0, 0.0, 90.0, 90.0), geom)").show

taken from from http://www.geomesa.org/documentation/user/spark/sparksql.html via spark`s more typesafe scala dataframe API?

Oliviervs · Accepted Answer · 2017-03-02 18:49:13Z

1

If you have created a function, you can register the created UDF using:

sparksession.sqlContext.udf.register(yourFunction)

I hope this helps.

answered Mar 2, 2017 at 18:49

Oliviervs

517 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Karan Gupta · Accepted Answer · 2019-07-21 23:18:20Z

0

Oliviervs I think he's looking for something different. I think Georg wants to use the udf by string in the select api of the dataframe. For example:

val squared = (s: Long) => {
  s * s
}
spark.udf.register("square", squared)

df.select(getUdf("square", col("num")).as("newColumn")) // something like this

Question in hand is if there exists a function called getUdf that could be utilized to retrieve a udf registered via string. Georg, Is that right?

answered Jul 21, 2019 at 23:18

Karan Gupta

616 bronze badges

1 Comment

Georg Heiler Over a year ago

;) actually I have a library (geospark) and that registers some functions, but discards / does not store the references returned by sparksession.sqlContext.udf.register and I wonder how to get them back, i.e. to not have to rely on random strings.

Collectives™ on Stack Overflow

use spark SQL udf in dataframe API

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related