fix: stitch multi-line stack traces into one log event in observability-logs-openobserve#281
Conversation
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@observability-logs-openobserve/helm/templates/fluent-bit/config.yaml`:
- Around line 89-90: Update the continuation regex in the “cont” rule to require
a word boundary or whitespace after “at” (for example, `at\s` or `\bat\b`),
while preserving the existing `--- End of` matching.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 016b99f0-4aee-41b8-ba3b-011b13be9a4f
📒 Files selected for processing (1)
observability-logs-openobserve/helm/templates/fluent-bit/config.yaml
| rule "start_state" "/^.*Exception[:,].*$/" "cont" | ||
| rule "cont" "/^\s+(at|--- End of).*$/" "cont" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add a word boundary after at in the continuation regex to prevent false-positive matches.
The cont rule /^\s+(at|--- End of).*$/ matches any line starting with whitespace followed by the literal at, including non-stack-frame lines like atmosphere or attached. Since this only applies after an exception start line, real-world risk is low — but adding at\s (or \bat\b) is a one-character fix that prevents incorrect log grouping in edge cases.
🛡️ Proposed fix
- rule "cont" "/^\s+(at|--- End of).*$/" "cont"
+ rule "cont" "/^\s+(at\s|--- End of).*$/" "cont"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| rule "start_state" "/^.*Exception[:,].*$/" "cont" | |
| rule "cont" "/^\s+(at|--- End of).*$/" "cont" | |
| rule "start_state" "/^.*Exception[:,].*$/" "cont" | |
| rule "cont" "/^\s+(at\s|--- End of).*$/" "cont" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@observability-logs-openobserve/helm/templates/fluent-bit/config.yaml` around
lines 89 - 90, Update the continuation regex in the “cont” rule to require a
word boundary or whitespace after “at” (for example, `at\s` or `\bat\b`), while
preserving the existing `--- End of` matching.
…ty-logs-openobserve The tail input only set 'multiline.parser docker, cri', which handles the container-runtime partial-line protocol but never concatenates application lines belonging to one stack trace, so each frame reached OpenObserve as its own record with its own inferred level. Add an application-level multiline filter (go, java, python built-ins plus a custom dotnet parser) between the tail input and the kubernetes filter, and document in parsers.conf how to extend the parser list for formats not covered by the built-ins. Signed-off-by: Tem Revil <temrevil@gmail.com>
TemRevil
force-pushed
the
fix/openobserve-multiline-stacktraces
branch
from
3e003b8 to
aa31194
Compare
1 participant
Purpose
Multi-line exceptions (a .NET
FluentValidation.ValidationException, or a Java/Go/Python stack trace) emitted by a workload are shipped to OpenObserve as one record per physical line instead of one record per exception, so a single error is scattered across many records and continuation lines get inconsistent log levels. Fixes openchoreo/openchoreo#4150.Approach
Follows the proposed fix in the issue, and mirrors #280 (the same fix for the opensearch module) so the sibling modules stay consistent:
multilinefilter between thetailinput and thekubernetesfilter, keyed onlogcontent, using the built-ingo, java, pythonparsers plus a customdotnetparser (buffer On,flush_ms 2000).dotnet[MULTILINE_PARSER]inparsers.conf— Fluent Bit has no built-in .NET parser. A comment documents how to add further custom parsers for formats the built-ins don't cover.Verification
dotnetrules against a representativeFluentValidation.ValidationExceptiontrace: the…Exception: …line matchesstart_stateonly, theat …/--- End of inner exception stack trace ---lines matchcontonly, and a following normal log line matches neither — so the exception is stitched into one record (level taken from the first line) and the next record starts cleanly.tail → multiline → kubernetes → http.Summary by CodeRabbit