Transforms – AWS EC2 Metadata – Support Additional IMDS Instance Metadata Categories

[commiterate]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Enrich events with data from IMDS to:

• Quickly locate impacted instances (e.g. partition, region, availability zone ID, host ID).
  • Placement categories such as partition (e.g. aws), region (e.g. us-east-1), availability zone ID (e.g. use1-az1, use1-az2), placement group, partition number, and host ID (for dedicated hosts) help determine impact locality.
  • Lifecycle properties such as EC2 Auto Scaling target lifecycle state help correlate on-instance events with external events.

Attempted Solutions

N/A

Proposal

The AWS EC2 Metadata transform currently supports a subset of IMDS instance metadata categories.

Support additional IMDS instance metadata categories. In particular:

• Services
  • AWS Partition (services/partition)
  • AWS Region Domain (services/domain)
• Placement
  • AWS Region (placement/region)
    • In multi-region deployments with global observability aggregation, it's useful to know an instance's region to identify region outages (e.g. us-east-1 is down but us-west-2 is up).
  • AWS Availability Zone ID (placement/availability-zone-id)
    • Across AWS accounts, the same availability zone name (e.g. us-east-1a, which the filter supports today) maps to different underlying availability zone IDs (e.g. use1-az1, use1-az2).
    • In multi-account deployments, it's useful to know an instance's availability zone ID to identify availability zone outages (e.g. use1-az1 is down but use1-az2 is up).
  • Placement Group Name (placement/group-name)
  • Placement Group Partition Number (placement/partition-number)
  • Dedicated Host ID (placement/host-id)
    • Useful for identifying bad dedicated hosts (e.g. EC2 Mac instances which require dedicated hosts due to Apple's EULA).
• IP Addresses (Public/Private IPv4/6)
  • The transform currently supports only the public + private IPv4 addresses.
• AWS EC2 Auto Scaling
  • Target Lifecycle State (autoscaling/target-lifecycle-state)
    • Useful for correlating EC2 Auto Scaling Lifecycle Hook events with on-instance events (e.g. graceful shutdown procedures).

References

• fluent-bit
  • Issue: Filters – AWS Metadata – Support Additional IMDS Instance Metadata Categories fluent/fluent-bit#9640
  • PR: filter_aws: add more IMDS categories fluent/fluent-bit#9727

Version

0.43.1