Unable load a CSV file as dataframe in spark

Question

I am trying to load a CSV file in the data frame and my objective is to display the first row as the column name of the CSV file. but while using the below code, I am getting the error

Exception in thread "main" java.lang.AbstractMethodError
    at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:278)

Code:

def main(args : Array[String]): Unit = {
 val spark : SparkSession = SparkSession
      .builder()
      .master("local")
      .appName("SparkSessioncsvExample")
      .config("spark.some.config.option", "some-value")
      .getOrCreate()
   val df = spark.read
      .format("csv")
      .option("header", "true") //reading the headers
      .load("D:/Scala/C2ImportCalEventSample.csv")}

But I'm able to load the file with the code:

val df = spark.sparkContext
        .textFile("D:/Scala/C2ImportCalEventSample1.csv")
        //.flatMap(header='true')
        .map(line => line.split(","))
        // .map(line => line.map()
        .toDF()

but in the second code file is getting successfully loaded but the first row is not getting as column_name of the data frame.

spark version is: spark-2.3.2  
scala 2.11.3  
jdk1.8.0_20
sbt-1.2.7

Thanks any anyone who can help me on this.

your second method is actually loading the text file.. so you wont get the column names.. could you please paste the sample csv in the question — stack0114106
– stack0114106, Commented Dec 31, 2018 at 18:57

Harjeet Kumar · Accepted Answer · 2019-01-01 06:56:43Z

1

java.lang.AbstractMethodError almost always means that you have different libraries on the classpath than at compilation time. In this case I would check to make sure you have the correct version of Scala (and only have one version of scala) on the classpath.

answered Jan 1, 2019 at 6:56

Harjeet Kumar

5242 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sanjeet Over a year ago

Thanks , Harjeet kumar, my env variable is pointing scala 2.12.8 and my sbt file it is 2.11.3,. I have change my env varibale to 2.11.8 version also in sbt file it is 2.11.8 version. and it s working file. Thanks for help.

Collectives™ on Stack Overflow

Unable load a CSV file as dataframe in spark

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related