Apache Spark Scala - data analysis - error

Question

I am new to/still learning Apache Spark/Scala. I am trying to analyze a dataset and have loaded the dataset into Scala. However, when I try to perform a basic analysis such as max, min or average, I get an error -

error: value select is not a member of org.apache.spark.rdd.RDD[Array[String]]

Could anyone please shed some light on this please? I am running Spark on the cloudlab of an organization.

Code:

// Reading in the csv file

val df = sc.textFile("/user/Spark/PortbankRTD.csv").map(x => x.split(","))  

// Select Max of Age

df.select(max($"age")).show()

Error:

<console>:40: error: value select is not a member of org.apache.spark.rdd.RDD[Array[String]]                                                
          df.select(max($"age")).show()

Please let me know if you need any more information. Thanks

To add to @thebluephantom's incredibly helpful comment, you should read about converting an RDD to a DataFrame. select is a method on DataFrame; you're handing it an RDD (as the error message indicates). — user4601931
– user4601931, Commented Apr 15, 2020 at 14:17

user4601931 · Accepted Answer · 2020-04-15 14:26:44Z

3

Following up on my comment, the textFile method returns an RDD[String]. select is a method on DataFrame. You will need to convert your RDD[String] into a DataFrame. You can do this in a number of ways. One example is

import spark.implicits._

val rdd = sc.textFile("/user/Spark/PortbankRTD.csv")
val df = rdd.toDF()

There are also built-in readers for many types of input files:

spark.read.csv("/user/Spark/PortbankRTD.csv")

returns a DataFrame immediately.

answered Apr 15, 2020 at 14:26

user4601931

5,3245 gold badges35 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

DLYT Over a year ago

Thank so much user4601931! Really appreciate the guidance. Will try this and revert.

user4601931 Over a year ago

No problem! If you think the answer addresses your question, you should click the checkmark to accept it. This way the question doesn't pop up again in the feed. This isn't definitive, and if a better answer comes along, you can accept that one instead.

Collectives™ on Stack Overflow

Apache Spark Scala - data analysis - error

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related