IMDS lookups retry every 15s forever when IMDS is unreachable from a container

[alexjurkiewicz]

Agent version

7.78.1

Bug Report

On EC2 instances with HttpPutResponseHopLimit: 1 and HttpTokens: required, IMDS is unreachable from the agent when it runs in a container. The hop limit drops the IMDSv2 token response before it reaches the container, and IMDSv1 is refused, so no code path can succeed.

cloud_hostinfo retries anyway, every 15s, indefinitely. Each attempt blocks a collector slot for ~600ms waiting on a socket, so roughly 4% of the time a collector slot is occupied doing nothing that can ever succeed.

Please back the IMDS lookup off to 60s after repeated failures.

Reproduction Steps

1. Run the agent in a container on an EC2 instance with HttpPutResponseHopLimit: 1 and HttpTokens: required.
2. agent status collector — cloud_hostinfo shows ~600ms average execution time, indefinitely.
3. From inside the container, curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' hangs until timeout, while curl http://169.254.169.254/latest/meta-data/instance-id returns 401 in under 10ms.

Other environment details

Seen on EKS Auto Mode, whose nodes default to those metadata options, but nothing here is specific to it — any container on a hop-limited instance should reproduce.

[jinh-labs]

@alexjurkiewicz Thanks for raising this. We are looking into it and will update this issue when we have more information.