How to add a step to emr cluster within a vpc through lambda function which is inside a same vpc and private subnet?

Question

when the lambda function is not within vpc then iam able to add a step to emr cluster but if the lambda function is residing inside vpc where emr cluster is present and same private VPC subnet also.

This time iam getting timeout error when iam trying to add a step in emr cluster using boto3 client module "add_job_flow_steps"

"errorMessage": "2020-05-14T02:48:46.771Z ad979ac2-ff26-476a-b301-23797caeeaa9 Task timed out after 123.10 seconds".

Do i need to add a VPC Endpoint for me to communicate between AWS services within same VPC Subnet or is there any other way which i could communicate?

Marcin · Accepted Answer · 2020-05-14 03:44:21Z

1

when the lambda function is not within vpc then iam able to add a step to emr cluster

This works because lambda not in vpc, can access internet. Subsequently, you can connect to public endpoint of AWS services, such as EMR.

if the lambda function is residing inside vpc where emr cluster is present and same private VPC subnet also.

This does not work, because lambda in VPC does not have internet access:

If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.

To enable your lambda access the EMR service you need to use either NAT gateway or VPC interface endpoint as shown in the following link:

Connect to Amazon EMR Using an Interface VPC Endpoint

Please not that lambda in VPC also requires modified execution role.

edited May 14, 2020 at 3:44

answered May 14, 2020 at 3:27

Marcin

241k16 gold badges315 silver badges368 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Abhishek Diwate Over a year ago

if we have lambda and emr in the same vpc private subnet . is there any possibility of direct communication without creating vpc end point

Marcin Over a year ago

@AbhishekDiwate You need to connect to AWS EMR service it self, which is not in your subnet. Thus you need the endpoints or nat gateway.

Abhishek Diwate Over a year ago

Hi Marcin, I'm getting this error when they are in same private subnet and same vpc .if they are in same private subnet and same vpc. VPC endpoint is not needed right??

Marcin Over a year ago

@AbhishekDiwate I would recommend testing with VPC endpoint for EMR. If this does not help, then there maybe some other issues.

Collectives™ on Stack Overflow

How to add a step to emr cluster within a vpc through lambda function which is inside a same vpc and private subnet?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related