Why does Spark application in Docker container fail with OutOfMemoryError: Java heap space?

Question

I'm using an r4.8xlarge on AWS Batch Service to run Spark. This is already a big machine, 32 vCPU, and 244 GB. On AWS Batch Service the process runs inside a Docker container. Out of multiple sources, I saw that we should use java with the parameters:

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1

Even with this parameters the process never when over 31Gb resident memory and 45 GB of virtual memory.

As analyzes I did:

java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 26.67G
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

second test

docker run -it --rm 650967531325.dkr.ecr.eu-west-1.amazonaws.com/java8_aws java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 26.67G
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

third test

java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=10 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 11.38G
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

The system is build with Native Packager as a standalone application. A SparkSession is built as follows with Cores equal to 31 (32-1):

SparkSession
  .builder()
  .appName(applicationName)
  .master(s"local[$Cores]")
  .config("spark.executor.memory", "3g")

Answer to egorlitvinenko:

$ docker stats
CONTAINER ID        NAME                                                                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
0c971993f830        ecs-marcos-BatchIntegration-DedupOrder-3-default-aab7fa93f0a6f1c86800   1946.34%            27.72GiB / 234.4GiB   11.83%              0B / 0B             72.9MB / 160kB      0
a5d6bf5522f6        ecs-agent                                                               0.19%               19.56MiB / 240.1GiB   0.01%               0B / 0B             25.7MB / 930kB      0

More tests, now with Oracle JDK, the memory never went over 4G:

$ 'spark-submit' '--class' 'integration.deduplication.DeduplicationApp' '--master' 'local[31]' '--executor-memory' '3G' '--driver-memory' '3G' '--conf' '-Xmx=150g' '/localName.jar' '--inPath' 's3a://dp-import-marcos-refined/platform-services/order/merged/*/*/*/*' '--outPath' 's3a://dp-import-marcos-refined/platform-services/order/deduplicated' '--jobName' 'DedupOrder' '--skuMappingPath' 's3a://dp-marcos-dwh/redshift/item_code_mapping'

I used the parameters -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 on my Spark, clearly not using all the memory. How can I go about this issue?

Where did you specify the memory of the driver? It's 1024M by default. How do you submit the Spark app? — Jacek Laskowski
– Jacek Laskowski, Commented Mar 29, 2018 at 10:23

Jacek Laskowski · Accepted Answer · 2018-03-29 10:34:17Z

1

tl;dr Use --driver-memory and --executor-memory while spark-submit your Spark application or set the proper memory settings of the JVM that hosts the Spark application.

The memory for the driver is by default 1024M which you can check out using spark-submit:

--driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).

The memory for the executor is by default 1G which you can check out again using spark-submit:

--executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G).

With that said, it does not really matter how much memory your execution environment has in total as a Spark application won't use more that the default 1G for the driver and executors.

Since you use local master URL the memory settings of the driver's JVM are already set when you execute your Spark application. It is simply too late to set the memory settings while creating a SparkSession. The single JVM of the Spark application (with the driver and a single executor all running on the same JVM) has already been up and so no config can change it.

In other words, how much memory a Docker container has has no impact on how much memory the Spark application use. They are environments configured independently. Of course, the more memory a Docker container has the more a process inside could ever have (so they are indeed interconnected).

Use --driver-memory and --executor-memory while spark-submit your Spark application or set the proper memory settings of the JVM that hosts the Spark application.

answered Mar 29, 2018 at 10:34

Jacek Laskowski

75k28 gold badges253 silver badges440 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

oleber Over a year ago

I tryed: 'spark-submit' '--class' 'integration.deduplication.DeduplicationApp' '--master' 'local[31]' '--executor-memory' '3G' '--driver-memory' '3G' '--conf' 'spark.executor.extraJavaOptions=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2' '/localName.jar' '--inPath' 's3a://dp-import-marcos-refined/platform-services/order/merged/*/*/*/*' '--outPath' 's3a://dp-import-marcos-refined/platform-services/order/deduplicated' '--jobName' 'DedupOrder' '--skuMappingPath' 's3a://dp-marcos-dwh/redshift/item_code_mapping' Didn't solve the issue

Jacek Laskowski Over a year ago

The following should be without quotes around 3G in '--executor-memory' '3G' '--driver-memory' '3G'.

oleber Over a year ago

spark-submit --class integration.deduplication.DeduplicationApp --master local[31] --executor-memory 3G --driver-memory 3G --conf -Xmx=150g /localName.jar --inPath s3a://dp-import-marcos-refined/platform-services/order/merged/*/*/*/* --outPath s3a://dp-import-marcos-refined/platform-services/order/deduplicated --jobName DedupOrder --skuMappingPath s3a://dp-marcos-dwh/redshift/item_code_mapping}}

changed nothing :(

Jacek Laskowski Over a year ago

What do you mean by "changed nothing"? Can you check out the memory settings in the Environment tab of the Spark app?

oleber Over a year ago

Batch service don't provide port mapping :(

|

Collectives™ on Stack Overflow

Why does Spark application in Docker container fail with OutOfMemoryError: Java heap space?

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related