Cannot create Dataframe in PySpark

Question

I want to create a Dataframe in PySpark with the following code

from pyspark.sql import *
from pyspark.sql.types import *

temp = Row("DESC", "ID")
temp1 = temp('Description1323', 123)

print temp1

schema = StructType([StructField("DESC", StringType(), False),
                     StructField("ID", IntegerType(), False)])

df = spark.createDataFrame(temp1, schema)

But i am receiving the following error:

TypeError: StructType can not accept object 'Description1323' in type type 'str'

Whats wrong with my code?

martinarroyo · Accepted Answer · 2018-10-01 07:20:26Z

4

The problem is that you are passing a Row where you should be passing a list of Rows. Try this:

from pyspark.sql import *
from pyspark.sql.types import *

temp = Row("DESC", "ID")
temp1 = temp('Description1323', 123)

print temp1

schema = StructType([StructField("DESC", StringType(), False),
                     StructField("ID", IntegerType(), False)])

df = spark.createDataFrame([temp1], schema)

df.show()

And the result:

+---------------+---+
|           DESC| ID|
+---------------+---+
|Description1323|123|
+---------------+---+

answered Oct 1, 2018 at 7:20

martinarroyo

9,7713 gold badges42 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Cannot create Dataframe in PySpark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related