Error in SqlContext import and parallelize Pyspark

Question

line = "Hello, world"
sc.parallelize(list(line)).collect()

I obtain the following error

TypeError: parallelize() missing 1 required positional argument: 'c'

I also have an other issue when creating a dataframe from a list of strings with only one column:

from pyspark.sql.types import *
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
schema = StructType([StructField("name", StringType(), True)])
df3 = sqlContext.createDataFrame(fuzzymatchIntro, schema)
df3.printSchema()

I obtain the following error:

----> 3 sqlContext = SQLContext(sc)
AttributeError: type object 'SparkContext' has no attribute '_jsc'

Thank you in advance

from pyspark.context import SparkContext from pyspark.sql.session import SparkSession sc = SparkContext spark = SparkSession.builder.appName("DFTest").getOrCreate() — A.Dorra
– A.Dorra, Commented Mar 19, 2018 at 12:25

Anahcolus · Accepted Answer · 2018-03-19 12:59:14Z

1

Looking at your comment above, you seem to have initialized sparkContext in a wrong way as you have done

from pyspark.context import SparkContext from pyspark.sql.session import SparkSession sc = SparkContext spark = SparkSession.builder.appName("DFTest").getOrCreate()

The correct way would be

from pyspark.sql.session import SparkSession
spark = SparkSession.builder.appName("DFTest").getOrCreate()
sc = spark.sparkContext

And spark object can do the work of sqlContext

answered Mar 19, 2018 at 12:59

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Koteeswaran nagarajan · Accepted Answer · 2022-05-09 10:02:40Z

0

I tried above suggestion in my pyspark in windows using the Jupyter terminal and it has worked. Please find my sample code below which worked for me

import findspark
findspark.init()
import pyspark
from pyspark.sql.session import SparkSession
spark = SparkSession.builder.appName("DFTest").getOrCreate()
sc = spark.sparkContext
words = sc.parallelize(["scala","java","hadoop","spark","akka","spark vs 
hadoop","pyspark","pyspark and spark"])
counts = words.count()
print("Number of elements in RDD -> %i" % (counts))

answered May 9, 2022 at 10:02

Koteeswaran nagarajan

111 bronze badge

1 Comment

ryanwebjackson Over a year ago

How is this different from Ramesh Maharjan's answer?

Collectives™ on Stack Overflow

Error in SqlContext import and parallelize Pyspark

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related