Mongodb Spark Connector An error occurred while calling o67.showString

Question

I am using below Python code for reading data from "MongoDB" in Spark and converting it DataFrame:

from pyspark.sql import SparkSession

# Initialize a Spark session
spark = SparkSession.builder \
    .appName("MongoDB Spark Connector Example") \
    .config("spark.mongodb.read.connection.uri", "mongodb://localhost:23017/") \
    .config("spark.mongodb.read.database", "db_name") \
    .config("spark.mongodb.read.collection", "coll_name") \
    .config("spark.sql.debug.maxToStringFields", 1000) \
    .getOrCreate()
df = spark.read.format("mongodb").load()
df.createOrReplaceTempView("temp")
sqlDf = spark.sql("SELECT id from temp")
sqlDf.show()

I am using:

Spark Version: 3.2.4
Mongo DB Version: 6
Scala Version: 2.12.15
Java 1.8
Python 3.8
Ubuntu 20.0.4
mongo-spark-connector_2.12:10.2.0

I am using below command to run above code

spark-submit  --packages org.mongodb.spark:mongo-spark-connector_2.12:10.2.0 test.py

I am getting the below error:

py4j.protocol.Py4JJavaError: An error occurred while calling o67.showString.
: java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/immutable/Seq;
    at com.mongodb.spark.sql.connector.schema.InternalRowToRowFunction.<init>(InternalRowToRowFunction.java:46)
    at com.mongodb.spark.sql.connector.schema.RowToBsonDocumentConverter.<init>(RowToBsonDocumentConverter.java:84)
    at com.mongodb.spark.sql.connector.read.MongoScanBuilder.<clinit>(MongoScanBuilder.java:72)
    at com.mongodb.spark.sql.connector.MongoTable.newScanBuilder(MongoTable.java:121)

Ali BOUHLEL · Accepted Answer · 2023-10-31 12:48:07Z

0

This Is probably related a version conflict between scala, pyspark and the mongodb driver you're using.

You can check the installed versions of spark and scala and search for the corresponding mongo dependency.

You can also try to uninstall everything and redo the installation by choosing the good versions from the start.

Check the spark official doc here for installation details if needed.

answered Oct 31, 2023 at 12:48

Ali BOUHLEL

6255 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Gaurav Srivastava Over a year ago

All version were correct but somehow the error was there, I tried the same thing inside virtual box and it worked. It means there was some version conflict in previous installation.

Ali BOUHLEL Over a year ago

You can uninstall everything and restart from scratch

Ali BOUHLEL Over a year ago

@GauravSrivastava do not hesitate to upvote and accept the answer if it was helpful. Thank you

Collectives™ on Stack Overflow

Mongodb Spark Connector An error occurred while calling o67.showString

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related