[Elasticsearch]: querylog Agent stream does not decode NDJSON, so events stay unparsed when the x-pack template wins

[consulthys]

Integration Name

Elasticsearch [packages/elasticsearch]

Dataset Name

elasticsearch.querylog

Summary

On ES 9.4+, query-log events shipped by the integration land as a JSON string in message. elasticsearch.querylog.* never populates, so the data view and dashboard are empty.

x-pack owns the destination template (logs-elasticsearch.querylog@template). The integration is only the shipper plus Kibana assets. The Agent stream is a plain logfile input. It never expands the NDJSON line. Parsing was left to pipeline-json.yml via index.default_pipeline on the Fleet template (priority 200). That template loses to the x-pack template (priority 240 after elastic/elasticsearch#158188, 250 before). The ingest pipeline never runs.

Filebeat already does this correctly. The elasticsearch querylog fileset uses filestream + ndjson (target: "", expand_keys: true). Its ingest pipeline only sets event.ingested / event.created.

Related:

• Support: https://github.com/elastic/sdh-elasticsearch/issues/10223
• ES template priority: Lower logs-elasticsearch.querylog@template priority to avoid conflicts elasticsearch#158188
• Original fileset: [Elasticsearch]: Create new file set for querylog #13374

What needs to change

1. Decode on the Agent (required)

packages/elasticsearch/data_stream/querylog/agent/stream/log.yml.hbs

Today it only has paths, tags, allow_deprecated_use, and exclude_files. Add JSON expansion equivalent to Filebeat. Smallest change that stays on logfile (the package policy template has no filestream input):

json.keys_under_root: true
json.overwrite_keys: true
json.expand_keys: true
json.add_error_key: true

expand_keys matters. ES writes dotted ECS keys (elasticsearch.querylog.type, etc.).

Alternative: decode_json_fields on message with target: "", overwrite_keys: true, expand_keys: true. Same outcome.

Do not switch this stream to filestream in this bugfix. That would need a new input on the package policy template and is a follow-up.

2. Stop treating the ingest pipeline as load-bearing (optional but should do it)

packages/elasticsearch/data_stream/querylog/elasticsearch/ingest_pipeline/pipeline-json.yml is doing the Agent’s job today. After shipper-side decode, keep default.yml for event.ingested, event.created, and host.name / host.id from elasticsearch.node.*.

Slim pipeline-json.yml so it is a no-op when message is already gone or already structured. Keep the event.dataset != elasticsearch.querylog drop if we still want that guard.

3. Package metadata

• Bump packages/elasticsearch/manifest.yml (current 1.21.1).
• Changelog entry, type bugfix.
• querylog/manifest.yml description still says “Collect … using log input”. Fine to leave, or mention NDJSON decode.

4. Tests

• Pipeline tests in querylog/_dev/test/pipeline/ can stay. They still cover the remaining pipeline.
• System test (querylog/_dev/test/system/test-default-config.yml) should assert elasticsearch.querylog.type (or similar) is present on ingested docs.
• elastic-package stacks often do not install logs-elasticsearch.querylog@template. A passing system test there does not prove the 9.4+ conflict is fixed. The Agent-side decode is what makes the test representative.

What not to change

Do not try to make the Fleet index template win on priority. Mappings, LogsDB, and DSL stay in x-pack. Do not put querylog parsing in logs@custom.

Workaround until a release

On the Agent querylog stream, add the same json.* settings (or pipeline pointing at the installed logs-elasticsearch.querylog-* ingest pipeline). Request-level pipeline overrides index.default_pipeline.

Expected result

With query logging enabled and the elasticsearch integration shipping *_querylog.json, documents in logs-elasticsearch.querylog-* have elasticsearch.querylog.* populated even when the x-pack template is the one that matched. The query analytics dashboard works without a manual pipeline override.

[adcleared]

Not really my place but raising the possibility that the integration pipeline might be more preferable? This would couple it to stack version and effectively make the existing pipeline for querylog in the integration (which does exist), meaningless?
Sorry for the double comment

[consulthys]

@adcleared integration pipelines are not tied to stack versions, they have their own release cycle (right now the version is 1.22.0).

Since Filebeat parses the log lines as JSON (without an ingest pipeline), the Elastic Agent should mimic the same behavior.

[adcleared]

Just clarifying that logs-elasticsearch.querylog@template is pushed by x-pack, what I meant to say was doing this will effectively make the integration pipeline irrelevant, acknowledging it doesn't function right now and the logs-elasticsearch.querylog@template doesn't set it.

I don't pretend to know enough to push either way, but I can certainly see the case where not having an easily modifiable pipeline might be annoying down the road - that said, you could at least modify the processors from the integration side, but these can have different functions to pipelines. Just raising :)

[consulthys]

Thanks @adcleared appreciate your input!
I think this is the direction we're going in the PR.