How to pass variable from Makefile to Python script through docker-compose

Question

I have a Makefile, that runs a docker-compose, which has a container that executes a python script. I want to be able to pass a variable in the command-line to the Makefile and print it within the python script (testing.py).

My directory looks like:

main_folder:
  -docker-compose.yaml
  -Makefile
  -testing.py

I have tried with the following configuration. The Makefile is:

.PHONY: run run-prod stop stop-prod rm

run:
    WORKING_DAG=$(working_dag) docker-compose -f docker-compose.yml up -d --remove-orphans --build --force-recreate

The docker-compose is:

version: "3.7"
services:
  prepare_files:
    image: apache/airflow:1.10.14
    environment:
      WORKING_DAG: ${working_dag}
      PYTHONUNBUFFERED: 1
    entrypoint: /bin/bash
    command: -c "python3 testing.py $$WORKING_DAG"

And the file testing.py is:

import sys

print(sys.argv[0], flush=True)

When I run in the command line:

 make working_dag=testing run

It doesn't fail but it does not print anything neither. How could I make it? Thanks

Rshad Zhran · Accepted Answer · 2021-02-03 09:42:16Z

I believe that the variable WORKING_DAG is getting assigned correctly through the command-line and Makefile is passing it correctly to the docker-compose. I verified it by running the container to not be destroyed and then after logging into the container, I checked the value of WORKING_DAG:

To not destroy the container once the docker execution is completed, I modified the docker-compose.yml, as follows:

version: "3.7"
services:
  prepare_files:
    image: apache/airflow:1.10.14
    environment:
      WORKING_DAG: ${working_dag}
      PYTHONUNBUFFERED: 1
    entrypoint: /bin/bash
    command: -c "python3 testing.py $$WORKING_DAG"
    command: -c "tail -f /dev/null"

airflow@d8dcb07c926a:/opt/airflow$ echo $WORKING_DAG
testing

The issue that docker does not display Python's std.out when deploying with docker-compose was already commented in Github, here, and it still not resolved. Making it work when using docker-compose, is only possible if we transfer/mount the file into the container, or if we use Dockerfile instead.

When using a Dockerfile, you only have to run the corresponding script as follows,

CMD ["python", "-u", "testing.py", "$WORKING_DAG"]

To mount the script into the container, please look at @DazWilkin's answer, here.

DazWilkin · Accepted Answer · 2021-02-02 17:29:01Z

You'll need to mount your testing.py into the container (using volumes). In the following, your current working directory (${PWD}) is used and testing.py is mounted in the container's root directory:

version: "3.7"
services:
  prepare_files:
    image: apache/airflow:1.10.14
    volumes:
      - ${PWD}/testing.py:/testing.py
    environment:
      PYTHONUNBUFFERED: 1
    entrypoint: /bin/bash
    command: -c "python3 /testing.py ${WORKING_DAG}"

NOTE There's no need to include WORKING_DAG in the service definition as it's exposed to the Docker Compose environment by your Makefile. Setting it as you did, overwrites it with "" (empty string) because ${working_dag} was your original environment variable but you remapped this to WORKING_DAG in your Makefile run step.

And

import sys

print(sys.argv[0:], flush=True)

Then:

make --always-make working_dag=Freddie run
WORKING_DAG=Freddie docker-compose --file=./docker-compose.yaml up
Recreating 66014039_prepare_files_1 ... done
Attaching to 66014039_prepare_files_1
prepare_files_1  | ['/testing.py', 'Freddie']
66014039_prepare_files_1 exited with code 0

Collectives™ on Stack Overflow

How to pass variable from Makefile to Python script through docker-compose

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related