Transforming a list into pyspark dataframe

Question

Using pyspark 2.1,I am trying to find some predicted result and the code is given below

restultSet=testingData.map(lambda p: (p.label, linearModel.predict(p.features))).collect()

Now the output of restultSet is in list and it looks like below

[(2.0, array([ 2.09078012])), (2.0, array([ 2.09078012])), (2.0, array([ 2.09078012])), (1.0, array([ 2.09078012])), (2.0, array([ 2.09078012])), (1.0, array([ 2.09078012]))]

When I am have give type(restultSet) it is showing me below result

I am struggling to convert from list to dataframe

I tried to use below snippets it didn't work . Please help

restultSet.toDF()

desertnaut · Accepted Answer · 2017-10-19 11:55:23Z

2

You cannot convert restultSet to a Spark dataframe because, due to collect, it is a Python list, and toDF works for RDDs.

Removing collect, and adding one more map to convert your numpy arrays to Vectors.dense should do the job.

Here is an example with the data you have provided:

import numpy as np
from pyspark.ml.linalg import Vectors

# your data as an RDD (i.e. before 'collect')
dd = sc.parallelize([(2.0, np.array([ 2.09078012])), (2.0, np.array([ 2.09078012])), (2.0, np.array([ 2.09078012])), (1.0, np.array([ 2.09078012])), (2.0, np.array([ 2.09078012])), (1.0, np.array([ 2.09078012]))])
dd.take(1)
# [(2.0, array([ 2.09078012]))]

df = dd.map(lambda x: (x[0], Vectors.dense(x[1]))).toDF()
df.show()
# +---+------------+ 
# | _1|          _2|
# +---+------------+
# |2.0|[2.09078012]| 
# |2.0|[2.09078012]|
# |2.0|[2.09078012]|
# |1.0|[2.09078012]|
# |2.0|[2.09078012]|
# |1.0|[2.09078012]|
# +---+------------+

To give names to the resulting columns, include them as a list argument in toDF, i.e. toDF(["column_1", "column_2"]).

edited Oct 19, 2017 at 11:55

answered Oct 19, 2017 at 11:38

desertnaut

60.8k32 gold badges155 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mrsrinivas Over a year ago

My intention was .toDf on List(Not on Rdd). As it will comes only with implicits in scala. Anyways thanks for pointing out.

Collectives™ on Stack Overflow

Transforming a list into pyspark dataframe

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related