EMR Spark deploy mode when using Docker

Question

I am deploying a spark job in AWS EMR and packaging all my dependencies using docker. My pythonized spark submit command looks like this

    ...
    cmd = (
            f"spark-submit --deploy-mode cluster "
            f"spark-submit --deploy-mode {deploy_mode} "
            f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker "
            f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE={docker_image} "
            f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG={config} "
            f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro "
            f"--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker "
            f"--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE={docker_image} "
            f"--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG={config} "
            f"--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro "
            f"{path}"
        )
    ...

It worked as expected when my deploy_mode is cluster but I don't see any of my docker dependency when deploy_mode is client. Can anyone help why this is happening and is it normal?

Ikfon10 · Accepted Answer · 2022-11-06 18:41:06Z

2

The docker containers are managed by Yarn on EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-docker.html

In client mode, your Spark driver doesn't run in a docker container because that process is not managed by Yarn, it is directly executed on the node that runs the spark-submit command. In cluster mode your driver is managed by Yarn and as so executed inside a docker container.

answered Nov 6, 2022 at 18:41

Ikfon10

363 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

EMR Spark deploy mode when using Docker

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related