fix: stitch multi-line stack traces into one log event in observability-logs-opensearch#280
fix: stitch multi-line stack traces into one log event in observability-logs-opensearch#280SAY-5 wants to merge 1 commit into
Conversation
…ty-logs-opensearch Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
📝 WalkthroughWalkthroughFluent Bit now assembles multiline Kubernetes application logs using built-in Go, Java, Python, and custom .NET parsers. The configuration adds a multiline filter and regex rules for .NET exception stack traces. ChangesMultiline log parsing
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@observability-logs-opensearch/helm/templates/fluent-bit/config.yaml`:
- Around line 93-94: Update the Fluent Bit multiline rules for the .NET parser:
adjust the continuation pattern in the “cont” rule to match “--- End of stack
trace from previous location where exception was thrown ---” at column 0, and
refine the “start_state” exception pattern so lines beginning with “--->
System...” remain in the current stack-trace group instead of starting a new
one.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d31974dc-7645-4dae-a5c0-f2560b028d59
📒 Files selected for processing (1)
observability-logs-opensearch/helm/templates/fluent-bit/config.yaml
| rule "start_state" "/^.*Exception[:,].*$/" "cont" | ||
| rule "cont" "/^\s+(at|--- End of).*$/" "cont" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for any existing .NET log samples or test fixtures in the repo
rg -i "End of.*stack trace|inner exception|--->" --type yaml --type markdown -C2Repository: openchoreo/community-modules
Length of output: 166
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file and nearby context.
wc -l observability-logs-opensearch/helm/templates/fluent-bit/config.yaml
sed -n '80,110p' observability-logs-opensearch/helm/templates/fluent-bit/config.yaml
# Look for any .NET stack trace samples or related parser rules anywhere in the repo.
rg -n -i --hidden --glob '!**/.git/**' \
'End of stack trace|inner exception|--->|multiline parser|dotnet|Exception[:,]' \
.
# List candidate docs/tests that may contain log samples.
git ls-files | rg -n '\.(md|mdx|yaml|yml|txt|json)$'Repository: openchoreo/community-modules
Length of output: 21888
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the fluent-bit config file in a small window around the parser.
cat -n observability-logs-opensearch/helm/templates/fluent-bit/config.yaml | sed -n '88,100p'
# Search for any .NET stack trace examples or related parser tests/docs in the repo.
rg -n -i --hidden --glob '!**/.git/**' \
'End of stack trace|inner exception|--->|dotnet|Exception[:,]' \
.
# Enumerate text fixtures and docs that might contain sample logs.
git ls-files | rg '\.(md|mdx|yaml|yml|txt|json)$'Repository: openchoreo/community-modules
Length of output: 20073
🌐 Web query:
.NET stack trace format inner exception prefix " ---> " end of stack trace from previous location where exception was thrown column 0 documentation
💡 Result:
In.NET, the message --- End of stack trace from previous location where exception was thrown --- is a marker inserted into an exception's stack trace to indicate that the exception was re-thrown or captured and re-thrown across asynchronous or thread boundaries [1][2]. When you see this marker, it signifies that the.NET runtime has merged two distinct call stacks into a single trace [2]. This frequently occurs when using async/await, where an exception is captured by ExceptionDispatchInfo (EDI) and then re-thrown to preserve the original stack trace information, despite the code continuing to execute on a different stack frame [3][4]. Key points regarding this format: 1. Purpose: It documents the transition point where an exception was moved from one execution context to another, ensuring developers can still trace the original source of the exception even after it has been re-thrown [3][4]. 2. Technical Origin: This marker was introduced to support better debugging of asynchronous code, as it allows the stack trace to span across the gap created by awaited tasks [3][5]. 3. Not an Inner Exception: It is important to distinguish this from an InnerException [6]. An InnerException represents a separate exception object that caused the current exception to be thrown [6]. The stack trace marker, by contrast, is purely a metadata annotation within the same exception object's stack trace string, indicating how that specific exception traveled through the call stack [3][2]. To view the full context of an error, including nested InnerExceptions, it is recommended to examine the Exception.ToString output, which recursively formats the exception, its message, its stack trace (including these markers), and its InnerException(s) [7][8].
Citations:
- 1: https://blog.elmah.io/understanding-net-stack-traces-a-guide-for-developers/
- 2: https://www.ageofascent.com/2018/01/26/stack-trace-for-exceptions-in-dotnet-core-2.1/
- 3: Does EDI message provide benefit? "End of stack trace from previous location..." dotnet/runtime#9517
- 4: https://gaevoy.com/2024/01/29/exception-rethrow-dotnet-framework-traps.html
- 5: Improved formatting for async stack traces dotnet/runtime#4996
- 6: https://learn.microsoft.com/en-us/dotnet/api/system.exception.innerexception?view=net-8.0
- 7: https://stackoverflow.com/questions/5928976/what-is-the-proper-way-to-display-the-full-innerexception
- 8: https://stackoverflow.com/questions/4272579/how-to-print-full-stack-trace-in-exception
.NET multiline parser still splits common stack traces
--- End of stack trace from previous location where exception was thrown ---is a column-0 marker, so^\s+(at|--- End of)misses it.---> System.NullReferenceException: ...is part of the same exception string, and^.*Exception[:,].*$will start a new group on it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@observability-logs-opensearch/helm/templates/fluent-bit/config.yaml` around
lines 93 - 94, Update the Fluent Bit multiline rules for the .NET parser: adjust
the continuation pattern in the “cont” rule to match “--- End of stack trace
from previous location where exception was thrown ---” at column 0, and refine
the “start_state” exception pattern so lines beginning with “---> System...”
remain in the current stack-trace group instead of starting a new one.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
3 participants
Purpose
Multi-line exceptions (a .NET
FluentValidation.ValidationException, or a Java/Go/Python stack trace) emitted by a workload are indexed into OpenSearch as one document per physical line instead of one document per exception, so a single error is scattered across manycontainer-logs-*documents and continuation lines get inconsistent log levels. Fixes openchoreo/openchoreo#4149.Approach
The
tailinput only usesmultiline.parser docker, cri, which handles the container-runtime partial-line protocol but does not concatenate application lines belonging to one stack trace. I added an application-levelmultilinefilter (keyed onlog) that runs before thekubernetesfilter, using Fluent Bit's built-ingo,javaandpythonstack-trace parsers..NET has no built-in Fluent Bit parser, so
dotnetis defined as a custom[MULTILINE_PARSER]inparsers.conf(matching theException:header andat .../--- End of ... ---continuation lines), and a comment documents how to add another custom parser for formats the built-ins do not cover.I validated the rendered ConfigMap with
fluent-bit:latest: the config loads without error and a sampleFluentValidation.ValidationExceptionwith itsat ...frames is emitted as a single record instead of separate documents.Related Issues
Resolves openchoreo/openchoreo#4149 (sub-issue of openchoreo/openchoreo#2974).
Checklist
Remarks
This is a config-only change to the fluent-bit ConfigMap template; the sibling
observability-logs-openobserveandobservability-logs-opensearch-based modules track the same problem under openchoreo/openchoreo#4150 and openchoreo/openchoreo#4151.Summary by CodeRabbit