EC2 instance being terminated after setting up lifecycle hooks

Question

I'm using the AWS CDK to add lifecycle hooks to my ALB in order to be notified by email when an instance gets terminated or when a new one is started. This is what my code looks like:

topic = sns.Topic(my_stack, "fleet_changes")
topic_hook = autoscaling_hooktargets.TopicHook(topic)

asg.add_lifecycle_hook(
    'instance-terminating-hook',
    lifecycle_transition=autoscaling.LifecycleTransition.INSTANCE_TERMINATING,
    lifecycle_hook_name="instance-terminating",
    notification_target=topic_hook,
    notification_metadata="INFO: An instance has been terminated"
)

asg.add_lifecycle_hook(
    'instance-launching-hook',
    lifecycle_transition=autoscaling.LifecycleTransition.INSTANCE_LAUNCHING,
    lifecycle_hook_name="instance-launching",
    notification_target=topic_hook,
    notification_metadata="INFO: A new instance has been launched"
)

This works fine as I'm indeed getting notifications when an instance is terminated or launched. The problem is that the instances are terminated every hour, and this stops happening when I remove the lifecycle hooks. This is what I see in the ALB events when the hooks are set:

At 2024-03-22T17:11:34Z an instance was taken out of service in response to a launch failure.

And right after:

At 2024-03-22T17:12:35Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2.

I've checked the application logs in Cloudwatch and haven't found anything that would indicate a bug, and there is currently little to no activity on these instances.

Any idea what could be the culprit?

Thank you.

Some helpful diagnostic steps here.

jarmod
– jarmod

2024-03-22 17:38:42 +00:00
Commented Mar 22, 2024 at 17:38 — jarmod
– jarmod, Commented Mar 22, 2024 at 17:38

kichik · Accepted Answer · 2024-04-09 11:36:57Z

0

You have to complete the lifecycle action after getting it. Use something like the following when getting it:

aws autoscaling complete-lifecycle-action --lifecycle-action-result CONTINUE \
  --lifecycle-hook-name my-launch-hook --auto-scaling-group-name my-asg \
  --lifecycle-action-token bcd2f1b8-9a78-44d3-8a7a-4dd07d7cf635

You can also put it in UserData to automate it. You would have to give your insurance permission to do so in its instance profile.

    aws autoscaling complete-lifecycle-action --lifecycle-action-result CONTINUE \
  --instance-id i-1a2b3c4d --lifecycle-hook-name my-launch-hook \
  --auto-scaling-group-name my-asg

answered Apr 9, 2024 at 11:36

kichik

34.9k7 gold badges102 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1319182 Over a year ago

Thanks for your replies. Now, that's a bit confusing because I did a few things since then. First I set the default action to CONTINUE, then reduced the heartbeat to 30 seconds. This backfired very badly because the instances would keeping cycling and I really don't know why. I ended up just removing the lifecycle for instance start-up, which works fine so far. I still don't understand why I was getting the previous outcomes.

Collectives™ on Stack Overflow

EC2 instance being terminated after setting up lifecycle hooks

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related