Creating and array from a txt file using spark shell, scala in particular

Question

I need some help with a transformation I want to make to a txt file for a homework in scala using command line of spark. Here is the array I created by using a DF:

scala> tempDF.collect()
res6: Array[org.apache.spark.sql.Row] = Array([bid|gender|department], [1|M|Informatics], [2|M|Low], [3|M|BusinessAdministration], [5|M|Mathematics], [6|M|Low], [7|M|Economics], [8|M|Economics], [9|M|Economics], [10|M|Economics], [11|M|Informatics], [13|M|Physics], [14|M|Informatics], [15|M|Informatics], [16|M|Economics], [17|M|Informatics], [18|M|Economics], [19|M|BusinessAdministration], [20|M|Mathematics], [21|M|Mathematics], [22|M|Economics], [23|M|Economics], [24|M|BusinessAdministration], [25|M|Informatics], [26|M|Statistics], [27|M|BusinessAdministration], [28|M|Economics], [29|M|Physics], [30|M|Physics], [31|M|Informatics], [32|M|Mathematics], [33|M|Economics], [34|M|BusinessAdministration], [35|M|Economics], [36|M|BusinessAdministration], [37|M|Mathema...

notNull · Accepted Answer · 2020-11-30 02:06:41Z

2

Spark supports reading delimited files using .csv method.

Try by reading csv file with delimiter specified as shown below.

spark.
read.
option("header","true").
option("delimiter","|").
csv("<txt file path>").
show()
//+---+------+-----------+
//|bid|gender| department|
//+---+------+-----------+
//|  1|     M|Informatics|
//|  2|     M|        Low|
//+---+------+-----------+

answered Nov 30, 2020 at 2:06

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Giannis Stas Over a year ago

Many many thx ! Now can I save it as an RDD and then DF ?

notNull Over a year ago

Not sure if i understand your question correctly, if you want to save as csv file then use df.write.csv("<path>") please refer these links for all available options: docs.databricks.com/data/data-sources/read-csv.html

mck · Accepted Answer · 2020-11-30 06:48:47Z

0

Write the dataframe in csv format, specifying that you want a header:

tempDF.write.format("csv").option("header", "true").save("file.txt")

answered Nov 30, 2020 at 6:48

mck

42.7k13 gold badges44 silver badges62 bronze badges

Collectives™ on Stack Overflow

Creating and array from a txt file using spark shell, scala in particular

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related