create pyspark dataframe from nested list

Question

I need to create a dataframe from nested list

I have tried different methods, But none worked

R = Row("id","age","serial")
List=[[1,2,3],[4,5,6],[7,8,9]]
sp=spark.createDataFrame([R(i) for i in (List)])

Expected:

noufel13 · Accepted Answer · 2019-08-28 13:50:31Z

1

Instead of R(i) you must use R(*i). This passes individual elements of the inner list to the Row object.

In addition to this, zip must be applied on the input list to get a list of tuples, like below,

[(1, 4, 7), (2, 5, 8), (3, 6, 9)]

Full code,

R = Row("id","age","serial")
L=[[1,2,3],[4,5,6],[7,8,9]]
sp=spark.createDataFrame([R(*i) for i in zip(*L)])
sp.show()

Output:

+---+---+------+
| id|age|serial|
+---+---+------+
|  1|  4|     7|
|  2|  5|     8|
|  3|  6|     9|
+---+---+------+

answered Aug 28, 2019 at 13:29

noufel13

6634 silver badges4 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

He is asking just for the transpose of your output.

My bad. Must've overlooked it. I've updated my answer. Thanks.