Convert String to Pyspark Dataframe

Question

I have one string in List something like

ListofString = ['Column1,Column2,Column3,\nCol1Value1,Col2Value1,Col3Value1,\nCol1Value2,Col2Value2,Col3Value2']

How do i convert this string to pyspark Dataframe like below

'\n' being a new row

Column1         Column2         Column3
-----------------------------------------
Col1Value1      Col2Value1      Col3Value1
Col1Value2      Col2Value2      Col3Value2

marc_s · Accepted Answer · 2020-01-17 05:46:03Z

3

You simply need to convert the list of string in the correct format like this:

# convert the list of string into proper format
>>> l = ' '.join(ListofString)
>>> l = l.replace(',',' ')
>>> l = [x.strip().split(' ') for x in l.split('\n')]

>>> print(l)

>>> [['Column1', 'Column2', 'Column3'], ['Col1Value1', 'Col2Value1', 'Col3Value1'], ['Col1Value2', 'Col2Value2', 'Col3Value2']]

>>> df = spark.createDataFrame(l[1:],l[0])

>>> df.show()

+----------+----------+----------+
|   Column1|   Column2|   Column3|
+----------+----------+----------+
|Col1Value1|Col2Value1|Col3Value1|
|Col1Value2|Col2Value2|Col3Value2|
+----------+----------+----------+

edited Jan 17, 2020 at 5:46

marc_s

760k186 gold badges1.4k silver badges1.5k bronze badges

answered Jan 15, 2020 at 20:39

YOLO

22k5 gold badges25 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert String to Pyspark Dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related