How to convert scala spark.sql.dataFrame to Pandas data frame

Question

I wanted to Convert scala dataframe into pandas data frame

    val collection = spark.read.sqlDB(config)
    collection.show()

    #Should be like df=collection

It would be easier if you use pyspark: stackoverflow.com/questions/50958721/… — Shaido
– Shaido, Commented Aug 5, 2019 at 8:03

Selnay · Accepted Answer · 2019-08-05 15:10:55Z

1

You are asking for a way of using a Python library from Scala. This is a bit weird to me. Are you sure you have to do that? Maybe you know that, but Scala DataFrames have a good API that will probably give you the functionality you need from pandas.

If you still need to use pandas, I would suggest you to write the data that you need to a file (a csv, for example). Then, using a Python application you can load that file into a pandas dataframe and work from there.

Trying to create a pandas object from Scala is probably overcomplicating things (and I am not sure it is currently possible).

answered Aug 5, 2019 at 15:10

Selnay

7114 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ravi · Accepted Answer · 2019-08-05 06:47:21Z

0

I think If you want to use pandas based API in SPARK code, then you can install Koalas-Python library. So, Whatever the function you want to use from pandas API directly you can embed them in SPARK code.

To install kolas

pip install koalas

answered Aug 5, 2019 at 6:47

Ravi

4701 gold badge3 silver badges17 bronze badges

2 Comments

Ravi Over a year ago

I think here collection variable is dataframe. toPandas() def would be there. If you apply the toPandas() function it will return pandas based data frame. This link will give more information about installing kolas and how to use It. medium.com/future-vision/…

Data_Insight Over a year ago

Not related to my question.

Collectives™ on Stack Overflow

How to convert scala spark.sql.dataFrame to Pandas data frame

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related