Calling JVM-based algorithms / functions from pySpark possible?

Question

I created a set of algorithms and helpers in Scala for Spark working with different formats of measured data. They are all based on Hadoop's FileInputFormat. I also created some helpers to ease working with time series data from a Cassandra database. I now need some advanced functions which are already present in Thunder, plus some of my colleagues who are to work with these helper functions want to use Python. Is it somehow possible to use these helper functions from python or do I have to reimplement them?

I read through a lot of docs and only found that you can load extra jars with pyspark, but not how to use the functions.

@eliasah It depends, doesn't it? You can trigger high level transformations but it is not possible to the same thing from the worker. — zero323
– zero323, Commented Feb 24, 2016 at 13:19
That's true ! I was thinking of the other way around like what I did here — eliasah
– eliasah, Commented Feb 24, 2016 at 13:26
So, if I created the "sc.coolMeasuringDataFile" via an implicit class, can I use that from pyspark and if yes, how do I do that? — rabejens
– rabejens, Commented Feb 24, 2016 at 14:05

rabejens · Accepted Answer · 2016-02-24 16:45:51Z

3

"By accident" I found the solution: It is the "Java Gateway". This is not documented in the Spark documentation (at least I didn't find it).

Here is how it works, using a "GregorianCalendar" as an example

j = sc._gateway.jvm
cal = j.java.util.GregorianCalendar()
print cal.getTimeInMillis()

However, passing the SparkContext does not work directly. The Java SparkContext is in the _jsc field:

ref = j.java.util.concurrent.atomic.AtomicReference()
ref.set(sc)

this fails. However:

ref = j.java.util.concurrent.atomic.AtomicReference()
ref.set(sc._jsc)

works.

However note that sc._jsc returns a Java-based Spark Context, i.e., a JavaSparkContext. To get the original Scala SparkContext, you have to use:

sc._jsc.sc()

edited Feb 24, 2016 at 16:45

answered Feb 24, 2016 at 16:23

rabejens

8,25213 gold badges65 silver badges123 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

eliasah Over a year ago

Good one ! Nevertheless it isn't documented in Spark because it's not Spark related by rather Java/Python interoperability related

Collectives™ on Stack Overflow

Calling JVM-based algorithms / functions from pySpark possible?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related