Not able to ping the AWS IoT VPC interface endpoint via EC2 in same subnet

Question

I am trying to create an AWS IoT VPC interface endpoint. I am using terraform. Each AWS account comes with an IoT Core endpoint like this: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com

They can be pinged from public internet.

Here is the code I am trying: Created a VPC with Internet gateway, Route tables and public subnets (easier to test, later will do it in private subnet). Created a security group which has allows incoming traffic on port 443, 80, 8883, 8 (ICMP) for CIDR "172.16.0.0/16". Outgoing allowed on all ports and all IP's.

Endpoint generating code:

resource "aws_vpc_endpoint" "iot_core_vpc_interface_endpoint" {
  vpc_id = aws_vpc.main.id
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [for subnet in aws_subnet.public_web_subnets : subnet.id]
  service_name        = "com.amazonaws.us-east-1.iot.data"
    
  private_dns_enabled = false
  security_group_ids  = [aws_security_group.endpoint_security_group.id]
}

For private hosted zone

data "aws_iot_endpoint" "iot_endpoint" {
  endpoint_type = "iot:Data-ATS"
}
    
resource "aws_route53_zone" "iot_private_hosted_zone" {
  name = data.aws_iot_endpoint.iot_endpoint.endpoint_address
    
  vpc {
    vpc_id = aws_vpc.main.id
    vpc_region = var.region
  }
}

Finally the Route 53 settings.

resource "aws_route53_record" "www" {
  zone_id = aws_route53_zone.iot_private_hosted_zone.zone_id
  name    = data.aws_iot_endpoint.iot_endpoint.endpoint_address
  type    = "A"
    
  alias {
    name = aws_vpc_endpoint.iot_core_vpc_interface_endpoint.dns_entry[0].dns_name
    zone_id = aws_vpc_endpoint.iot_core_vpc_interface_endpoint.dns_entry[0].hosted_zone_id
    evaluate_target_health = true
  }
}

I created an EC2 instance in the same public subnet, attached the same security group and tried to ping the endpoint xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com from it but it gave no answer.

dig xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com

gave the private IPs associated with the VPC endpoint.

When I remove the Route 53 hosted zone and the records, I am able to ping the endpoint. I guess it just starts sending traffic over internet rather than the VPC Endpoint.

I think I am missing out on some small but important detail here.

More details: Thanks to response of @wikeegan I tried this from my EC2 instance and got this:

sh-5.2$ curl -v xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com:8883
* Host xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com:8883 was resolved.
* IPv6: (none)
* IPv4: 172.16.1.249, 172.16.2.91, 172.16.3.221
*   Trying 172.16.1.249:8883...
* Connected to xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com (172.16.1.249) port 8883
> GET / HTTP/1.1
> Host: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com:8883
> User-Agent: curl/8.5.0
> Accept: */*

I am trying to come up with a simple explanation as to what is happening.

I am not an expert on this. I will read through the link. The link I posted said: > You must manually create DNS records in a private hosted zone that is attached to your VPC. > For Enable DNS name, make sure that Enable for this endpoint is not selected. Neither AWS IoT Core data plane nor AWS IoT Core credential provider supports private DNS names yet. — NotAgain
– NotAgain, Commented Apr 5, 2024 at 14:02
Yeah, disregard my previous comment. I think the document you linked says you need to create alias records for VPC Endpoint interfaces: Within your private hosted zone, create an alias record for each elastic network interface IP for the VPC endpoint.. Are they the same as the endpoint address of the IoT endpoint? — Marko E
– Marko E, Commented Apr 5, 2024 at 14:03
Yes. As per the documents I put the end point address in the hosted zone as well as in the record name. > The DNS name must be your domain configuration name or your IoT:Data-ATS endpoint. An example DNS name can be: xxx-ats.data.iot.region.amazonaws.com > For Record name, enter iot:Data-ATS endpoint. — NotAgain
– NotAgain, Commented Apr 5, 2024 at 14:07
Why not just set private_dns_enabled = true on the endpoint, and let AWS manage associating that address with your VPC? When you do that, you don't need to create a Route53 private hosted zone or any DNS records yourself. Also, make sure you have enable_dns_support = true on your VPC. — Mark B
– Mark B, Commented Apr 5, 2024 at 14:24
As per documentation "Neither AWS IoT Core data plane nor AWS IoT Core credential provider supports private DNS names yet.". — NotAgain
– NotAgain, Commented Apr 5, 2024 at 14:34

wikeegan · Accepted Answer · 2024-04-05 21:04:02Z

1

Oddly enough been working on the same thing, trying to save $ on NAT costs. While struggling with this for most the day, figured it had to deal something with routing or firewalls, since the ats endpoint resolves to the private IP of VPC endpoint, and I could verify the ping request went out of the subnet using flow logs.

On a whim, I tried both telnet and curl to test the connection to the ats endpoint, and both seemed to work when connecting to port 443 only.

curl -v {id}-ats.iot.eu-west-1.amazonaws.com -p 443

telnet {id}-ats.iot.eu-west-1.amazonaws.com 443

Verified my lambda functions are also able to connect to the IoT endpoints for publishing messages. Appears to be an issue with the way we were testing the connection.

answered Apr 5, 2024 at 21:04

wikeegan

112 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

NotAgain Over a year ago

Did you try with port 8883? The MQTT port?

Collectives™ on Stack Overflow

Not able to ping the AWS IoT VPC interface endpoint via EC2 in same subnet

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related