Convert Array[Row] to DataFrame in Spark/Scala

Question

I want to convert Array[org.apache.spark.sql.Row] to a DataFrame. Could anyone suggest me a better way?

I tried to first convert it into RDD and then tried to convert it into Dataframe , but when I perform any operation on the DataFrame , exceptions are shown.

val arrayOfRows = myDataFrame.collect().map(t => myfun(t))
val distDataRDD = sc.parallelize(arrayOfRows)
val newDataframe = sqlContext.createDataFrame(distDataRDD,myschema)

Here myfun() is a function which returns Row (org.apache.spark.sql.Row). The contents in the array is correct and I am able to print it without any problem.

But when I tried to count the records in the RDD, it gave me the count as well as a warning that one of the stage contains a task of very large size.I guess I am doing something wrong. Please help.

Hussein Hazimeh · Accepted Answer · 2016-06-28 00:16:56Z

1

You have a bug in the first line. collect returns an Array while map is a method that operates on DataFrames/RDDs.

Try val arrayOfRows = myDataFrame.map(t => myfun(t)).collect() instead.

answered Jun 28, 2016 at 0:16

Hussein Hazimeh

111 bronze badge

Sign up to request clarification or add additional context in comments.

3 Comments

rvp Over a year ago

I am getting this error when i change the order org.apache.spark.SparkException: Task not serializable

Hussein Hazimeh Over a year ago

arrayofRows is actually of type DataFrame so there's no need for lines 2 and 3 (sc.parallelize accepts RDDs and not DataFrames, which is the reason behind the new exception)

rvp Over a year ago

I am getting the error as soon as I enter the first line val arrayOfRows = myDataFrame.collect().map(t => myfun(t))

Omkar Neogi · Accepted Answer · 2019-01-15 06:53:10Z

1

case class PgRnk (userId : Long , pageRank: Double ) 
// create a case class 

sc.parallelize(pg10.map(r1 => PgRnk(r1.getLong(0), r1.getDouble(1)))).toDS() 
// convert into a dataset, sc.parallelize converts the array into a RDD, and then to DS

edited Jan 15, 2019 at 6:53

Omkar Neogi

7852 gold badges12 silver badges31 bronze badges

answered Mar 20, 2017 at 18:56

Ashish Markanday

1,4101 gold badge11 silver badges11 bronze badges

Collectives™ on Stack Overflow

Convert Array[Row] to DataFrame in Spark/Scala

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related