2

I've attempted to create an Azure Function to run dbt (data build tool), although can't get the Function to work when deployed (works fine locally), as getting a permissions error.

Overview of Az Function set-up:

  • OS – Linux
  • Publishing as Code, not Docker – is this an issue?
  • Plan - Functions Premium EP1
  • Python 3.9
  • dbt-core==1.3.0 added to requirements.txt
  • Testing via Azure portal Function 'Code + Test'

At this stage I'd just like the dbt helper text to appear in the logging.info output as test that dbt can be called.

__init.py__ contains the following:

import logging
import subprocess

import azure.functions as func

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    logging.info('Start With Popen dbt')
    #with subprocess.Popen(["ls"], stdout=subprocess.PIPE, shell=True) as proc: # this runs fine, although shell cmd shell=True 
    with subprocess.Popen(["/tmp", "dbt"], stdout=subprocess.PIPE) as proc: # Attempt to output dbt helper as doesn't write to log- When running locally subprocess.Popen(["dbt"] remove "/tmp"
        for line in proc.stdout:
            line = line.decode('utf-8').replace('\n', '').strip()
            logging.info(line)
    logging.info('End With Popen dbt')

Error message:

Result: Failure Exception: PermissionError: [Errno 13] Permission denied:'/tmp' Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", 
line 458, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.9/concurrent/futures/thread.py", 
line 58, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", 
line 701, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/extension.py", 
line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger_dbt/__init__.py", 
line 11, in main with subprocess.Popen(["/tmp", "dbt"], stdout=subprocess.PIPE) as proc: # Attempt to output dbt helper as doesn't write to log- When running locally remove subprocess.Popen(["dbt"] remove "/tmp" File "/usr/local/lib/python3.9/subprocess.py", 
line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", 
line 1821, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)

Within __init.py__ running ls instead of dbt works fine, although this is a shell cmd as opposed to a function from an installed package.

I've seen setting WEBSITE_RUN_FROM_PACKAGE = 0 mentioned when investigating the issue and also had a response to the following MS Q&A Post suggesting that - Although when setting to 0 I can't seem to redeploy.

I have confirmed the WEBSITE_RUN_FROM_PACKAGE has not been set; by not set there is no setting associated WEBSITE_RUN_FROM_PACKAGE as opposed to 0/1, by looking at:

  • Azure > Function App > Advanced Tool - Environment > AppSettings
  • zure \> Function App \> Configuration \> App Setting
  • VS Code \> Function App \> App Setting

I'm unsure as what the issue is, either:

  • I can't achieve what I want by running dbt in an Az Fucntion as calling a function from a package.
  • I'm missing something from my set-up/config
  • WEBSITE_RUN_FROM_PACKAGE = 0 should work and I have a seperate deployemnt issue?
  • I need to publish as Docker instead of Code?

I've created the following test repo if required for reference: azfunc-dbt-test - the bicep file contains the Function Config settings, although will not work if you attempt to deploy - still trying to get to grips with Bicep.

Any help or suggestions with this issue would be much appreciated.

Update

One thing I've noticed within Azure Portal > Function App > Development Tools > SSH If I run pip list dbt does not appear, only the following:

pip        22.0.4
ptvsd      4.3.2
setuptools 58.1.0
wheel      0.38.4

Next I ran find -name "*dbt*" Files of interest are:

./site/wwwroot/.python_packages/lib/site-packages/bin/dbt
./site/wwwroot/.python_packages/lib/site-packages/dbt
./site/wwwroot/logs/dbt.log

Running: dbt results in -bash: dbt: command not found pip install dbt-core then dbt results in the dbt helper text as expected

This seems to install dbt into a different location:

find -name "*dbt*"
/usr/local/bin/dbt
./usr/local/lib/python3.9/site-packages/dbt
1
  • subprocess.Popen(["/tmp", "dbt"]) is similar to typing /tmp dbt at your shell. The error is saying you can't execute the /tmp directory. If dbt is installed (globally or in an active venv), you should just be able to subprocess.Popen(["dbt"], shell=True), or use the preferred subprocess.run(["dbt"], shell=True) Commented Jan 18, 2023 at 17:05

1 Answer 1

0

After a lot of pain and trial and error(!!!) I went down the Azure Function Publish as Docker route, as opposed to Code as was my original attempt.

I then set up an Azure DevOps Pipeline.

Within the Pipeline script azure-pipelines.yml I noticed that a target path was being set for pip install - pip install --target="./.python_packages/lib/site-packages" - I assume this is also the publish as Code default setting.

- bash: |
        pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

Removing the target path for pip install solved my issue and allowed me to run the dbt package, assume the same result for any additional packages installed.

- bash: |
    pip install -r ./requirements.txt
  workingDirectory: $(workingDirectory)
  displayName: 'Install application dependencies'

I assume the issue is that the dbt package could not be accessed in the default installation location.

I'm unsure if the default pip install path can be altered when publishing an Azure Function as Code - Hopefully someone can respond to this.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.