I'm trying to use an autoloader to pick up files from a container in ADLS and store them as delta files in a delta table explicitly defined at an external location in a different container within the same storage account called "Bronze". When i try to write the stream with Trigger = Once option, i get an error " com.databricks.sql.cloudfiles.errors.CloudFilesException: Failed to create an Event Grid subscription.Please make sure that your service principal has 'write' permissions
The exact Error is as follows :
{ "error" : { "code" : "LinkedAuthorizationFailed", "message" : "The client '910adb4f-a9dd-4666-ace1-xxxxxxxxxx' with object id '910adb4f-a9dd-4666-ace1-xxxxxxxxxx' has permission to perform action 'Microsoft.EventGrid/eventSubscriptions/write' on scope '/subscriptions/485bf1a4-b8c5-48af-ba04-xxxxxxxxx/resourceGroups/StocksDev/providers/Microsoft.Storage/storageAccounts/stockpricesdev/providers/Microsoft.EventGrid/eventSubscriptions/databricks-query-8030d5e2-7058-4cfd-aa7f-b20867633d45-source-0'; however, it does not have permission to perform action(s) 'Microsoft.Storage/storageAccounts/write' on the linked scope(s) '/subscriptions/485bf1a4-b8c5-48af-ba04-xxxxxxxxxx/resourcegroups/StocksDev/providers/Microsoft.Storage/storageAccounts/stockpricesdev' (respectively) or the linked scope(s) are invalid." } }
I have now double checked permissions for the service principal and its according to docs:
Storage blob data contributor on storage account - stockpricesdev
Storage Queue Data contributor (I'm using the latest runtime > 8.0 without connection string so i guess i have added it just incase although not required)
EventGrid EventSubscription Contributor on the resource group - StockDev
I can also see the inherited role assignments within scopes. I'm not sure why this is happening as a contributor role would be sufficient to perform write action
