Convert list of lists to pyspark dataframe?

Question

Having trouble converting the following list to a pyspark dataframe.

lst = [[1, 'A', 'aa'], [2, 'B', 'bb'], [3, 'C', 'cc']]

cols = ['col1', 'col2', 'col3']

Desired output:

    +----------+----------+----------+ 
    | col1     | col2     | col3     |
    +----------+----------+----------+ 
    | 1        | A        | aa       |
    +----------+----------+----------+ 
    | 2        | B        | bb       |
    +----------+----------+----------+ 
    | 3        | C        | cc       |
    +----------+----------+----------+

I'm essentially looking for the pandas equivalent of:

df = pd.DataFrame(data=lst,columns=cols)

vht981230 · Accepted Answer · 2022-05-24 22:58:01Z

3

If you have pandas package installed then can just import the dataframe to pyspark using spark.createDataFrame

import pandas as pd
from pyspark.sql import SparkSession


lst = [[1, 'A', 'aa'], [2, 'B', 'bb'], [3, 'C', 'cc']]
cols = ['col1', 'col2', 'col3']

df = pd.DataFrame(data=lst,columns=cols)

#Create PySpark SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("spark") \
    .getOrCreate()

#Create PySpark DataFrame from Pandas
sparkDF=spark.createDataFrame(df) 
sparkDF.printSchema()
sparkDF.show()

Alternatively, you can also do it without having pandas

from pyspark.sql import SparkSession

lst = [[1, 'A', 'aa'], [2, 'B', 'bb'], [3, 'C', 'cc']]
cols = ['col1', 'col2', 'col3']

#Create PySpark SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("spark") \
    .getOrCreate()

df = spark.createDataFrame(lst).toDF(*cols)
df.printSchema()
df.show()

edited May 24, 2022 at 22:58

answered May 24, 2022 at 22:51

vht981230

5,3152 gold badges11 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

chicagobeast12 Over a year ago

I was getting an error - most likely datatype related since spark is very particular about datatypes. That's why I was looking for a pyspark based solution (ie. the list in the question is an example)

vht981230 Over a year ago

@chicagobeast12 Could you share the error message? I've also posted an approach that doesn't require pandas. Does that help resolve the issue?

Collectives™ on Stack Overflow

Convert list of lists to pyspark dataframe?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related