Working on my free trial azure account , I am trying to copy csv files to ADLS Gen2 and save the dataframe as table in adls silver layer.
code: DForderItems = spark.read.csv("abfss://[email protected]/retailfiles/orderItems.csv",header=False,schema=schema) ** I am able to read csv file into DForderitems , but the trickiest part is i am unable to save it as table in given path as below.
DForderItems**.write.option("path","abfss://[email protected]/retailfiles/orderItems").option("mergedschema", True).mode("append").saveAsTable("retail.orderItems") ** Error : ** [NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] No parent external location was found for path 'abfss://[email protected]/retailfiles/orderItems'. Please create an external location on one of the parent paths and then retry the query or command again. File , line 2
- I tried creating a table in external location using sql
%sql CREATE EXTERNAL LOCATION silv_layer URL 'abfss://[email protected]/retailfiles/' WITH (CREDENTIAL (STORAGE_ACCOUNT_KEY = 'GlGLL4o2tXZUawz0CqVgguhKGsAN2YLQRIUs56yw8PHTw8zYQIWc2+gWFojXFWWo/puH/Q2e/t6B+AStOAyWig=='));
Still got this error : [PARSE_SYNTAX_ERROR] Syntax error at or near 'LOCATION'. SQLSTATE: 42601
- I tried creating a database and writing table into it
#spark.sql
("CREATE DATABASE IF NOT EXISTS retail LOCATION 'abfss://[email protected]/retailfilessilver'") DForderItems.write.option("path","abfss://[email protected]/retailfiles/orderItems").option("mergeschema", True).mode("append").saveAsTable("retail.orderItems")
But i got another error [NO_PARENT_EXTERNAL_LOCATION_FOR_PATH] No parent external location was found for path 'abfss://[email protected]/retailfiles/orderItems'. Please create an external location on one of the parent paths and then retry the query or command again. File , line 2
