Setup scripts in Azure DevOps MS hosted container pipeline

Question

I need to compile a CMake based C++ project using GCC. It depends on MKL, and for successful Cmake configuration, compilation and test execution, I need run the following comands beforehand

        source /opt/intel/bin/compilervars.sh -arch intel64
        export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

In order to run this with Azure pipelines I have a container which I am able to run based on the documentation from https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops.

Normally the above mentioned setup scripts would be called during container startup (https://hub.docker.com/layers/vvtk/vvcoreazurelinuxdockeragent/latest/images/sha256-c5e3775546ee90a111c9ae700306eb4cd1ebc710686bda5011633c4e5e883e13?context=repo) however it seems (as also described in https://stackoverflow.com/a/63643979/15128314 this startup CMD command is not execute since Azure pipelines do not actually call docker run

As a result I am forced to replicate this into multiple steps of my pipeline job (basically each of config, build and test), since these env vars are also not persistent across the different steps. How can I solve this more efficiently? The pipeline looks horrible ..

  - script: |
      source /opt/intel/bin/compilervars.sh -arch intel64
      export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
      (more cmds here)
    displayName: config_Linux_x64_Release

  - script: |
      source /opt/intel/bin/compilervars.sh -arch intel64
      export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
      (more cmds here)
    displayName: build_Linux_x64_Release
  
  - script: | 
        source /opt/intel/bin/compilervars.sh -arch intel64
        export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
        (more cmds here)
    displayName: test_Linux_x64_Release

Hi Did you get a chance to check out below answer? How did it go? — Levi Lu-MSFT
– Levi Lu-MSFT, Commented Feb 11, 2021 at 4:20

Levi Lu-MSFT · Accepted Answer · 2021-02-16 07:20:48Z

0

You can try using the logging command(ie. ##vso[task.setvariable]) to set the system variable and avoid the replication for export command. See below:

You can define environment variable in the pipeline Variable section like below:

variables:
  LD_LIBRARY_PATH: /usr/local/lib:$LD_LIBRARY_PATH

Or you can try adding a script task in the top of your pipeline to run below command:

- script: |
  
    #below script will only take effect in the following tasks. 
      
    source /opt/intel/bin/compilervars.sh -arch intel64
    echo "##vso[task.setvariable variable=LD_LIBRARY_PATH]/usr/local/lib:$LD_LIBRARY_PATH" 

  displayName: 'SetVariable'

Note: the variable set up in above script task only takes effect in the following task.

Please refer to this thread.

Update:

For source command only applies for current shell, and each script step will launch a new shell. So you will still need to repeat source commands in each script step.

Or you can run config, build, test in a single script step to avoid replication.

edited Feb 16, 2021 at 7:20

answered Feb 8, 2021 at 7:39

Levi Lu-MSFT

30.8k3 gold badges42 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

iv1 Over a year ago

this would only work for LD_LIBRARY_PATH but does not persist whatever source /opt/intel/bin/compilervars.sh -arch intel64 actually sets, so no it does not completely solve this issue. Similarly for Virtual environments with python, see also medium.com/faun/…

iv1 Over a year ago

@multipitch according to stackoverflow.com/a/60664697/15128314 you were able to use virtual environments across different steps. I cannot really make this work, see also medium.com/faun/… - did this really solve your issue?

Levi Lu-MSFT Over a year ago

@iv1 I am afraid you have to repeat source command in each script step. This is because each script step will launch a new shell, and previous shell will be closed.

iv1 Over a year ago

ok I see - thanks anyways! I wanted to avoid clubbing everything into the same step, as I need to check success for each of them and abort execution if necessary

Collectives™ on Stack Overflow

Setup scripts in Azure DevOps MS hosted container pipeline

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related