TypeError: sql() missing 1 required positional argument: 'sqlQuery' in pyspark

Question

I'm trying to estimate proportion of maths marks in pyspark 3.0.1 on data bricks. Some of the cases my total_marks are 0.So I wrote following code

 df_data.registerTempTable('myTable')
 df_oversees2=SQLContext.sql("select A.*,case when total_marks=0 then 0 else (maths_marks/total_marks) end as prop_maths from myTable A ")

But I'm getting error message

TypeError: sql() missing 1 required positional argument: 'sqlQuery'

Can you please help me to resolve the issue? I also verified that my total_marks & maths_marks are present in my table using below code

df1 = sqlContext.sql("select maths_marks,total_marks from myTable")
df1.show()

I'm seeing the output for both the fields

mck · Accepted Answer · 2021-02-15 12:13:53Z

5

SQLContext is different from sqlContext - the latter is an SQLContext attached to a SparkContext, while the former is a class not attached to any SparkContext. Therefore, you need to do:

sqlContext = SQLContext(sc)
df_oversees2 = sqlContext.sql("your query")

In any case, it is deprecated and the better way would be to call

spark.sql("your query")

where spark is a sparkSession object.

edited Feb 15, 2021 at 12:13

answered Feb 15, 2021 at 12:06

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JDoe Over a year ago

I'm getting parse exception with spark.sql

JDoe Over a year ago

I have resolved the issue. By mistake there was a "#" given. Removed that & now it's running

Collectives™ on Stack Overflow

TypeError: sql() missing 1 required positional argument: 'sqlQuery' in pyspark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related