does 1 dpu setting change when I use glue python shell instead of glue spark?. I recently saw a post Maximum number of concurrent tasks in 1 DPU in AWS Glue and I saw they were talking about glue spark, but not from aws glue python shell, that's why I have that doubt.
1 Answer
As per this doc an AWS Glue job of type Python shell can be allocated either 1 DPU or 0.0625 DPU.By default, AWS Glue allocates 0.0625 DPU to each Python shell job.
You are billed $0.44 per DPU-Hour in increments of 1 second, rounded up to the nearest second, with a 1-minute minimum duration for each job of type Python shell.
A single DPU provides processing capacity that consists of 4 vCPUs of compute and 16 GB of memory as per this.
At max you can provide 1 DPU as it doesn't need many resources because there is no distributed processing involved. You try giving more than 1 DPU and will encounter error as shown below:
5 Comments
jaimeardpm83_
Hi! thanks for u answer, but i have still the doubt, 1dpu of glue spark equals 1 dpu of glue python (e.g "Each DPU is configured with 2 executors"), implies the same in both cases ?
Prabhakar Reddy
@Jaimeardp Yes it is exactly same as Glue DPU when it comes to capacity except that there will not be any executors as these will be launched only when you run a Glue job.
Prabhakar Reddy
you can consider this as an. EC2 machine with 4 vCPUs and 16 GB RAM with pre-installed python and supported libraries.
jaimeardpm83_
This much is clear with python shell :D, then glue job (spark) instead of just only instance, it would be a cluster ?
Prabhakar Reddy
yep it requires a minimum of two DPUs and maximum that meets your requirement.