2

I am trying to run a Data Pipelin in Azure Devops with the following YAML definition

This is requirements.txt file:

trigger: none

schedules:
  - cron: "0 0 * * *"
    displayName: "Daily midnight data pipeline run"
    branches:
      include:
      - master
    always: true

name: 'data_pipeline'
jobs:
  - job: 'data_pipeline_job'
    pool:
      vmImage: 'ubuntu-16.04'
    variables:
      - group: KeyVault
      - group: ProductionEnvVars
    steps:
      - task: UsePythonVersion@0
        inputs:
          versionSpec: '3.9.0'
          architecture: 'x64'
    
      - script: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
        displayName: 'Install requirements'
    
      - script: |
          python src/my_custom_package/create_data.py
        displayName: 'Create and Register Data'
        env:
          STORAGE_ACCT_NAME: $(StorageAccountName)
          STORAGE_ACCT_KEY: $(StorageAccountKey)
          TENANT_ID: $(TenantID)
          SPN_ID: $(SpnID)
          SPN_PASSWORD: $(SpnPassword)
          AML_WORKSPACE_NAME: $(AmlWorkspaceName)
          RESOURCE_GROUP: $(ResourceGroup)
          SUBSCRIPTION_ID: $(SubscriptionID)

However after some successful prerequisites installs then I start getting these errors

 Using legacy 'setup.py install' for numpy, since package 'wheel' is not installed.
  Installing collected packagesGenerating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/1afa3c52-aff1-4a34-ac26-f17b4946deec.sh
Collecting pip
  Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-20.2.4
Obtaining file:///home/vsts/work/1/s/src (from -r requirements.txt (line 8))
Collecting azureml-sdk==1.2.0
  Downloading azureml_sdk-1.2.0-py3-none-any.whl (4.6 kB)
Collecting azure-storage-blob==12.3.0
  Downloading azure_storage_blob-12.3.0-py2.py3-none-any.whl (279 kB)
Collecting joblib==0.14.1
  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)
Collecting pandas==1.0.3
  Downloading pandas-1.0.3.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/hostedtoolcache/Python/3.9.0/x64/bin/python /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_yuvh7a4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.13' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"''
       cwd: None
  Complete output (4485 lines):
  Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.7" and platform_system == "AIX"' don't match your environment
  Collecting setuptools
    Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
  Collecting wheel

1 Answer 1

2

Azure still is not compatible with 3.9. See also at https://github.com/numpy/numpy/issues/17482

Sign up to request clarification or add additional context in comments.

3 Comments

so? whats the fix?
1) Wait, 2) Switch back to 3.8.6. 3) Use numpy.whl from lfd.uci.edu/~gohlke/pythonlibs if it is possible.
3.8.6 fixed the issue

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.