I've attempted to create an Azure Function to run dbt (data build tool), although can't get the Function to work when deployed (works fine locally), as getting a permissions error.
Overview of Az Function set-up:
- OS – Linux
- Publishing as Code, not Docker – is this an issue?
- Plan - Functions Premium EP1
- Python 3.9
- dbt-core==1.3.0 added to requirements.txt
- Testing via Azure portal Function 'Code + Test'
At this stage I'd just like the dbt helper text to appear in the logging.info output as test that dbt can be called.
__init.py__ contains the following:
import logging
import subprocess
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
logging.info('Start With Popen dbt')
#with subprocess.Popen(["ls"], stdout=subprocess.PIPE, shell=True) as proc: # this runs fine, although shell cmd shell=True
with subprocess.Popen(["/tmp", "dbt"], stdout=subprocess.PIPE) as proc: # Attempt to output dbt helper as doesn't write to log- When running locally subprocess.Popen(["dbt"] remove "/tmp"
for line in proc.stdout:
line = line.decode('utf-8').replace('\n', '').strip()
logging.info(line)
logging.info('End With Popen dbt')
Error message:
Result: Failure Exception: PermissionError: [Errno 13] Permission denied:'/tmp' Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py",
line 458, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.9/concurrent/futures/thread.py",
line 58, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py",
line 701, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/extension.py",
line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger_dbt/__init__.py",
line 11, in main with subprocess.Popen(["/tmp", "dbt"], stdout=subprocess.PIPE) as proc: # Attempt to output dbt helper as doesn't write to log- When running locally remove subprocess.Popen(["dbt"] remove "/tmp" File "/usr/local/lib/python3.9/subprocess.py",
line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py",
line 1821, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)
Within __init.py__ running ls instead of dbt works fine, although this is a shell cmd as opposed to a function from an installed package.
I've seen setting WEBSITE_RUN_FROM_PACKAGE = 0 mentioned when investigating the issue and also had a response to the following MS Q&A Post suggesting that - Although when setting to 0 I can't seem to redeploy.
I have confirmed the WEBSITE_RUN_FROM_PACKAGE has not been set; by not set there is no setting associated WEBSITE_RUN_FROM_PACKAGE as opposed to 0/1, by looking at:
Azure > Function App > Advanced Tool - Environment > AppSettingszure \> Function App \> Configuration \> App SettingVS Code \> Function App \> App Setting
I'm unsure as what the issue is, either:
- I can't achieve what I want by running dbt in an Az Fucntion as calling a function from a package.
- I'm missing something from my set-up/config
WEBSITE_RUN_FROM_PACKAGE = 0should work and I have a seperate deployemnt issue?- I need to publish as Docker instead of Code?
I've created the following test repo if required for reference: azfunc-dbt-test - the bicep file contains the Function Config settings, although will not work if you attempt to deploy - still trying to get to grips with Bicep.
Any help or suggestions with this issue would be much appreciated.
Update
One thing I've noticed within Azure Portal > Function App > Development Tools > SSH If I run pip list dbt does not appear, only the following:
pip 22.0.4
ptvsd 4.3.2
setuptools 58.1.0
wheel 0.38.4
Next I ran find -name "*dbt*" Files of interest are:
./site/wwwroot/.python_packages/lib/site-packages/bin/dbt
./site/wwwroot/.python_packages/lib/site-packages/dbt
./site/wwwroot/logs/dbt.log
Running:
dbt results in -bash: dbt: command not found
pip install dbt-core then dbt results in the dbt helper text as expected
This seems to install dbt into a different location:
find -name "*dbt*"
/usr/local/bin/dbt
./usr/local/lib/python3.9/site-packages/dbt
subprocess.Popen(["/tmp", "dbt"])is similar to typing/tmp dbtat your shell. The error is saying you can't execute the/tmpdirectory. If dbt is installed (globally or in an active venv), you should just be able tosubprocess.Popen(["dbt"], shell=True), or use the preferredsubprocess.run(["dbt"], shell=True)