Add AWS OTel investigation templates (ALB, Lambda, RDS, SQS)

[JM-elastic]

Summary

Adds four observability root-cause-analysis templates for the AWS OTel (EDOT awscloudwatch) integrations currently in tech preview / heading to GA:

• aws-alb-5xx-investigation-otel — ALB 5xx surge (target vs ELB 5xx vs latency)
• aws-lambda-errors-investigation-otel — Lambda error spike (errors vs throttles vs duration)
• aws-rds-connection-exhaustion-investigation-otel — RDS connection exhaustion vs other DB bottlenecks + ALB cascade
• aws-sqs-consumer-lag-investigation-otel — SQS backlog (consumer lag vs stalled vs producer spike)

Each template:

• solutions: [observability], categories: [root-cause-analysis, monitoring], availability: \">=9.5.0\"
• Dual triggers: alert + manual (manual inputs for testing without an alert)
• Uses elasticsearch.esql.query + ai.classify + ai.summarize against default EDOT index patterns (metrics-aws.*.otel-*)
• No install.form — no connectors; index patterns stay in consts as EDOT defaults

Source: developed/proven against the Forge AWS FIS plane for the AWS OTel content workstream. Confirmed with @talboren that library PRs are welcome; CI validation is still incomplete so please help validate step types / runtime once reviewed.

Validation

npm run build:catalog (local, with env overrides):

> @elastic/workflows-library@0.0.0 build:catalog
> node scripts/build-catalog.mjs

[build-catalog] Using KIBANA_MAIN_VERSION override: 9.6.0
[build-catalog] Using KIBANA_NAMED_MINORS override: [9.5, 9.6]
[build-catalog] Loaded 22 template(s)
[build-catalog] Resolved main → Kibana 9.6.0
[build-catalog]   → v1/9.5/  (kibana 9.5.0, 22 templates)
[build-catalog]   → v1/9.6/  (kibana 9.6.0, 22 templates)
[build-catalog]   → v1/main/  (kibana 9.6.0, 22 templates)
[build-catalog]   → v1/templates/  (22 version-keyed YAML bodies)
[build-catalog] Done.

Test plan

• Catalog build green in CI (generator only today)
• Install each template in a 9.5+ Kibana with Template library enabled
• Manual run with sample inputs against a cluster that has metrics-aws.*.otel-* data (or confirm graceful on-failure: continue when empty)
• Confirm ai.classify / ai.summarize step types resolve on the target stack
• Spot-check alert trigger wiring (inputs expected from alert action context)

cc @talboren — as discussed, please help with validation since library CI validation isn't landed yet.

Made with Cursor

[talboren]

afaiu alert-triggered executions expose the alert payload under event (event.alerts), but everything below reads from inputs.*, which are only defined for the manual trigger. wouldn't the alert path run with empty resource names / timestamp (same for the other 3 templates)? should we derive these from event or remove the alert trigger for now?

[JM-elastic]

Confirmed and fixed — you were right that the alert path ran with empty inputs. Rather than guess field paths, we captured a real alert payload (probe workflow wired to a live rule): the useful fields are event.alerts[N].kibana.alert.grouping.<Dimension> (the rule's group-by values as structured fields — FunctionName, QueueName, DBInstanceIdentifier, and LoadBalancer arrives intact as the full app/<name>/<id> ref) plus kibana.alert.start for the timestamp.

Each workflow now starts with a resolve step: manual inputs.* take precedence, else the alert's grouping fields, else a sentinel that trips the evidence gate (see the other thread). Inputs are now required: false so alert runs pass validation.

Two wiring facts worth documenting (now in comments in the YAML): the alert trigger only fires when a rule carries the "Run Workflow" action, and that action should use summaryMode: false — with true one event can batch several alerts (in our capture, alerts[0] was a different team's RDS instance) and only the first would be investigated. Also, ES|QL rules can emit alerts for degenerate/null groups whose grouping has no resource field — those runs now end in an explicit insufficient_evidence refusal instead of investigating garbage.

Validated live: alert-triggered runs for all four workflows, fired by the shipped aws_*_otel rule templates against real injected incidents (Lambda error flood, SQS backpressure, RDS connection exhaustion, ALB target-5xx cascade).

[talboren]

we continue on failure for all the metrics queries, but then always pass their output to ai.classify. if an index is missing or a query fails, wouldn't we still force a potentially "convincing evidence" from empty data? wondering if the required queries should fail the workflow, and only the optional SLO/alert queries should continue.

[JM-elastic]

Agreed and restructured. Required evidence queries now hard-fail (no on-failure: continue), and — the part static review couldn't see — that alone isn't enough: a successful query over a wrong/missing resource returns one row of nulls, which the classifier happily pattern-matched to its healthy category (a false all-clear, verified live). So every workflow now has a deterministic evidence gate (if on documents_found > 0) in front of all AI steps; the else-branch emits a workflow.output with status: insufficient_evidence and refuses to diagnose.

One refinement to "required should fail": CloudWatch omits zero-valued sparse metrics, so fields like HTTPCode_ELB_5XX_Count are unmapped until the first such error ever lands — and ES|QL fails verification on unknown columns, killing the query even when the alert's own subject fields are present (we hit exactly this on a live target-5xx alert). So required steps reference only dense fields the triggering condition guarantees (RequestCount etc.), and each sparse family is its own optional phase-split step whose absence is treated as "no such errors observed". SLO/related-alert anchors stay optional as you suggested.

[talboren]

looks like the baseline and incident windows overlap between T-20m and T-12m. is that intentional? wouldn't incident data included in the baseline weaken the comparison?

[talboren]

|
V

[JM-elastic]

Fixed in all four (it was in ALB/RDS/SQS too, same consts): baseline is now T-50m→T-20m against incident T-20m→T+5m — contiguous, no overlap.

[talboren]

not sure on the logic itself, but something my agent flagged: why are we adding 1 to invocations here? e.g. 1 error out of 1 invocation would become a 50% error rate instead of 100%. can we handle zero invocations separately and otherwise divide by the actual count?

[JM-elastic]

Fixed: CASE(invocations > 0, TO_DOUBLE(errors) / TO_DOUBLE(invocations), null) in both windows, and the classifier is explicitly told a null rate means zero invocations, not 0%. Two related live findings while validating: (1) the queries were also silently mixing the Average/Sum/Maximum stat streams the collector ships (non-integer error counts) — all aggregations now carry per-agg WHERE attributes.stat == … filters, matching what the alert templates themselves do; (2) with correct math, the classifier initially flagged ambient drift (13% vs 11% baseline) as an incident, so the instructions now carry a materiality bar — re-tested healthy on a fleet with ~8% ambient error rate.

[talboren]

general comment: there's quite a bit of AWS/domain logic in the queries and classifier prompts here, and i'm not sure we can confidently validate all of it in a regular workflow review. can you run the 4 templates through an AWS/observability-focused agent (or get a domain review) and confirm the query outputs actually support each classification? if you already did that then sorry for the comment 😅 would be great to add the result to the validation section.

[talboren]

what happens with the output of this step when the workflow ends?

do we want a final workflow.output here? ai.summarize generates the actual investigation result, but afaiu it remains only a step output, so callers/composed workflows can't consume it as the workflow result. maybe we should expose steps.synthesize_findings.output.content?

[JM-elastic]

Added — each workflow now ends with a workflow.output step exposing status, resource, triggered_by, classification, rationale, and summary (and the insufficient-evidence branch emits its own), so callers and composed workflows get a real result. Validated on a 9.6.0 stack.

[talboren]

ran the new workflow validator against a release schema generated from my local 9.6.0 Kibana (i don't think our schemas changed since):

schema completeness gate passed (62 registered steps / 28 triggers all present)
4/4 templates passed the template schema, DAG, and Liquid syntax validation
so schema-wise these look good on 9.6.

[talboren]

ran the new workflow validator against a release schema generated from my local 9.6.0 Kibana (i don't think our schemas changed since):

schema completeness gate passed (62 registered steps / 28 triggers all present) 4/4 templates passed the template schema, DAG, and Liquid syntax validation so schema-wise these look good on 9.6.

schema validation is green, but it only checks the schema, DAG, and the Liquid syntax not whether references resolve or the steps actually execute. can you add actual run results for each template, including at least one alert-triggered run?

[talboren]

few comments

[JM-elastic]

Pushed f65459a addressing all review feedback — every change was validated live rather than just re-authored: real AWS fleet (EC2/ECS/RDS/SQS/Lambda/ALB) → EDOT awscloudwatch collector → Elastic 9.6.0, with the shipped aws_*_otel alerting rule templates instantiated and each workflow fired by its real alert against a real injected incident.

Validation matrix (all alert-triggered end-to-end unless noted):

Workflow	True positive	Negative / adversarial
RDS	connections-high alert → connection_exhaustion (73-conn plateau, ~500× baseline, other resources nominal)	freeable-memory alert → unrecognized, correctly refused its namesake diagnosis
Lambda	high-error-rate alert → 4 per-alert runs, resource resolved from grouping.FunctionName	volume-surge with flat error rate → refused elevated_errors; ambient drift → healthy
SQS	backlog-growth alert → consumer_lag (72k sent vs 27k deleted), anchored on a genuinely VIOLATED oldest-age SLO	—
ALB	shipped target-5xx alert → target_5xx_backend (5,451/5,513 requests), full DB→gateway-503→ALB-5xx cascade visible	sparse ELB-5xx absence read as "no LB-level errors", not as failure
all four	—	junk resource / empty inputs / degenerate null-group alerts / not-yet-ingested windows → deterministic insufficient_evidence, zero AI steps executed

One fix beyond the review threads: the ALB workflow queried metrics-aws.applicationelb.otel-*, but aws_elb_metrics_otel is a content package whose alert templates and dashboard all read metrics-aws.elb.otel-* — corrected to the canonical dataset.

Two calibration observations for the integrations packages (not changed here, just noting from the live runs): the Lambda high-error-rate default of 1% fires perpetually on any workload with a nontrivial ambient error rate, and the RDS connections > 100 default is unreachable on small instance classes (db.t3.micro ceilings at ~73) — both templates already carry "tune me" comments, this is evidence they're load-bearing.

🤖 Generated with Claude Code

[JM-elastic]

Two follow-ups for review honesty:

e6a3606 closes a classifier gap the live rounds exposed: the Lambda taxonomy had no volume-driven category, so a 13.5× invocation flood with a flat error rate — correctly refused as elevated_errors — fell through to duration_degradation on a +6% duration wiggle. Added load_surge (invocations ≫ baseline, rate near baseline, no throttles) with explicit precedence, plus a materiality bar on duration. Regression-tested against the recorded incidents: the flood window now classifies load_surge, the healthy window still classifies healthy.

ALB validation caveat: our lab account's ApplicationELB metrics are discovery-truncated (shared account, ~6.5k series vs the receiver's 500 limit), so the ALB end-to-end run rides a named-mode collector workaround that reshapes summary datapoints into the per-stat gauge form discovery mode produces natively. The workflow's queries were validated against that reshaped data and match what the aws_elb_metrics_otel alert templates read — but a customer environment with untruncated discovery is argued equivalent, not directly tested. The other three workflows were validated on plain discovery-mode data with no such caveat.

🤖 Generated with Claude Code

[JM-elastic]

Self-review (author-conducted, fresh adversarial pass)

I authored this PR, so this isn't independent review — flagging that up front; a human co-sign is still worth having. But here's a genuine critical pass, actively trying to break my own assumptions. Findings, most substantive first:

IMPORTANT — required queries hard-fail on transient errors, not just on missing data. The whole design removes on-failure: continue from the evidence queries so an empty/failed result can't feed a false all-clear. Correct for the empty case (the gate handles it). But it also means a transient ES timeout or a momentary cluster hiccup on, say, connections_incident aborts an otherwise-legitimate investigation. We traded "false confidence from empty data" for "a network blip kills the run." That's probably the right trade, but it's worth an explicit decision: should required steps carry a bounded retry (retry-then-fail) so a transient error doesn't look like a missing resource? Right now they don't.

IMPORTANT — self-relative baseline degrades for sustained incidents. Windows are baseline T−50m→T−20m vs incident T−20m→T+5m. That's clean for a recent onset, but an incident sustained past ~30 minutes poisons its own baseline (the "baseline" window already shows the degraded state), weakening every comparison. Same limitation the alert-rule side has. Not fixable without a longer-history baseline; worth a one-line caveat in the template so operators know onset-detection is the design point.

SUGGESTION — the ALB app_logs step queries logs-*.otel-*. Internally consistent with an all-OTel deployment, but a customer whose AWS logs arrive via the aws_logs integration has them in classic-shape logs-aws_logs.*, so the log layer silently returns nothing for them. It's on-failure: continue, so it degrades gracefully — but as written the app-log evidence only ever materializes on OTel-shipped logs. Consider querying both shapes, or documenting the assumption.

SUGGESTION — Liquid → ES|QL string interpolation. Resolved resource identifiers go straight into == "{{ ... }}". Alert-sourced values are AWS resource IDs (safe charset), but a manual input containing a \" would break the query into a confusing verification error rather than a clean message. Low severity (operator-supplied, trusted), noting for completeness.

Testing gap. The PR ships four customer-facing templates with no CI-runnable test — validation was live (extensively, see the run history), but there's no automated check that the YAML stays schema-valid or that the Liquid renders. If this repo has a template-validation harness, these should hook into it; if not, that's a broader gap worth a follow-up.

What holds up under scrutiny: the deterministic evidence gate ahead of every AI step, the sparse-metric-per-family handling (and the live-discovered no-healthy-targets absence signature), the per-stat aggregation filters, and the honesty of the scope caveats (ALB partially incident-validated, etc.). The live validation evidence is genuinely thorough.

Verdict: Comment (not a self-approval). Nothing here is merge-blocking, but the two IMPORTANT items deserve either a fix or an explicit "accepted, documented" before this ships to customers. I'd merge after those are addressed or consciously waived.

[talboren]

@JM-elastic is this ready to be reviewed again? the comments seems AI generated and i'm not sure whether it's ready to pick up again or not yet?

[JM-elastic]

@talboren thanks for checking in - yes I finished my last round of updates yesterday and hadn't gotten around to pinging you yet. Not sure if you'd prefer to keep the AI generated comments in or not (I left them so that others who may be using AI for review can have the extra context).

[talboren]

lgtm - 1 note: in alerting rules, summaryMode defaults to true, but these workflows only inspect event.alerts[0]. unless the user enables “Run per alert,” the remaining alerts are silently ignored. can we guard against multiple alerts or handle them instead of relying on a non-default action setting?

[talboren]

just noticed the 4 templates moved inputs to the workflow root level, they need to be under the manual trigger..

[JM-elastic]

4 screenshots provided below, showing example execution of each workflow

Lambda

SQS

ALB

RDS

cc @talboren