Skip to main content
44 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
0 votes
1 answer
49 views

I am using emr 6.15 and hudi 0.14 I submitted following hudi job which should create a database and a table in aws glue. IAM Role assigned to EMR serverless has all neccessary permissions of s3 and ...
Roobal Jindal's user avatar
  • 294
0 votes
0 answers
42 views

I have since switched from EMR on EC2 to EMR serverless. I used to use interactive notebooks with EMR on EC2. I am trying to use the EMR studio workspace (notebooks) with EMR serverless application ...
DirtyDan's user avatar
  • 1
0 votes
0 answers
99 views

I'm working with AWS EMR Serverless, and I need to construct a job URL for an EMR Serverless job to be sent in a message notification in case of state change. The desired URL includes the associated ...
user27008283's user avatar
  • 35
1 vote
0 answers
70 views

I’m working with a large transaction dataset (~1 billion rows) in PySpark on AWS EMR. My goal is to perform feature engineering where I compute statistics like sum, mean, standard deviation, and ...
Meriiiiii's user avatar
  • 11
0 votes
1 answer
206 views

I have a Docker image that contains some application code that interacts with Spark. Is it possible to submit this image to a Spark cluster for execution? If so, how? # Not a real command $ aws emr-...
sdgfsdh's user avatar
  • 37.8k
1 vote
1 answer
624 views

When EMR cluster is created, there is a provision to provide bootstrap actions as shown below. aws emr create-cluster --name "Test cluster" --release-label emr-7.1.0 --use-default-roles --...
Partha's user avatar
  • 580
0 votes
1 answer
843 views

I'm trying to get my head around the TaskFlow API & XCom in airflow and am getting stuck, hoping someone here can help. I'm using EmrServerlessCreateApplicationOperator and I want to pass a value ...
jamiet's user avatar
  • 12.7k
0 votes
1 answer
356 views

I'm seeing an inexplicable array index reference error, Index 1 out of bounds for length 1 ... which I can't explain because I don't see any relevant arrays being referenced in my context of an AWS ...
SomeDude's user avatar
  • 31
0 votes
1 answer
1k views

When I define an spark's executor's memory to 12gb, it actually allocates almost half of it like 6.7gb. Tried setting 20gb as well, then it allocates close to 11gb, half of it. I have defined ...
Roobal Jindal's user avatar
  • 294
0 votes
0 answers
396 views

I have been working on a request that extracts data from an ORACLE 19c instance, and that is processed using aws emr-serveless using spark jdbc connections. The big picture is that I can't connect to ...
eduardollopes's user avatar
  • 1
0 votes
1 answer
240 views

If I pass more than one subnet Id to EmrServerlessCreateApplicationOperator via the networkConfiguration attribute, I receive an error. If I use a single subnet Id the operator works fine. This is the ...
singleton's user avatar
  • 171
1 vote
2 answers
2k views

I am trying to create a emr-serverless application through the EmrServerlessCreateApplicationOperator but I keep facing the error botocore.exceptions.NoRegionError: You must specify a region. I am ...
Hasham's user avatar
  • 43
1 vote
1 answer
2k views

I'm new with EMR-serverless and I want to know how to pass, in a spark application, jar and packages as for example: spark-submit --deploy-mode client --jars /usr/lib/hudi/hudi-spark3.3-bundle_2.12-0....
Valle1208's user avatar
  • 43
1 vote
1 answer
3k views

Hi i am new to EMR serverless and trying to learn. I have a pyspark project which i want to run using EMR serverless. I tried using console but it is not letting me provide folder location as input. i ...
Corey A's user avatar
  • 11
0 votes
1 answer
281 views

I am new to AWS and my account got hacked and in order to secure the account I have been advised to delete IAM roles. There is one role called AWSServiceRoleForAmazonEMRServerless that I am unable to ...
Naren Nallapareddy's user avatar
  • 3
1 vote
1 answer
874 views

I want to launch a Spark job on EMR Serverless from Airflow. I want to use Spark 3.3.0 and Scala 2.13 but the 6.9.0 EMR Release ships with Scala 2.12. I created a FAT jar including all Spark ...
Oscar Drai's user avatar
  • 181
0 votes
2 answers
1k views

From boto3 doc for the start_job_run, it seems like I have to create job run every time I want to trigger a job. Does it really have to work that way? Can't I take the ID of the existing job, which ...
nirkov's user avatar
  • 829
1 vote
0 answers
2k views

I have been connecting with SQL Server using EMR Serverless App v-6.8.0 for Spark. So, I have tested code in local machine as well as on ec2 but when I ran the code on this serverless cluster I got an ...
Muhammad Ashir Ali's user avatar
  • 65
1 vote
2 answers
4k views

I have an EMR Serverless application that is getting stuck in executions timeouts for some reason. I have tested all s3 connections and it's working. The problem is happening during the execution of a ...
Renan Nogueira's user avatar
  • 165
1 vote
1 answer
4k views

I'm trying to run some jobs on aws cli using a virtual environment where I installed some libraries. I followed this guide; the same is here. But when I run the job I have this error: Job execution ...
solopiu's user avatar
  • 766
3 votes
0 answers
2k views

I'm trying to run a spark job using EMR Serverless but the issue is I cannot pass the list of jars and archives to the spark job. The spark properties section does not seem to allow passing in a comma ...
Philip K. Adetiloye's user avatar
  • 3,270
0 votes
1 answer
444 views

Based on the documentation, Amazon EMR serverless seems to accepts only Spark and Hive as job driver. Is there any support for custom Hadoop jar for map reduce jobs on serverless similar to EMR ?
Avik Das's user avatar
  • 131