fix(monitoring): tighten grafana log panel regex (ECHO-815)#22
Draft
spashii wants to merge 1 commit into
Draft
Conversation
Replace substring patterns like (?i)(error|exception|fail|fatal|critical|panic|warn|warning) with level-anchored \b(ERROR|FATAL|CRITICAL|Exception|Traceback)\b |\[(error|fatal|critical)\] across all per-service Error Logs panels, the Echo Application Error Logs panel, and the logfilter variable. Same shape for warning panels with WARN|WARNING. Underscore is a word character in RE2, so \berror\b does not match inside transcript_error / latest_error / etc. — drops the directus column noise without exclusion lists. Case-sensitive uppercase catches python logging level prefixes; bracketed lowercase catches nginx-style. Exception/Traceback covers python tracebacks. Real fix (promtail label extraction) supersedes this — separate follow-up. Refs: ECHO-815 Co-authored-by: Sameer <sameer@dembrane.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the substring false-positive problem on Grafana log panels described in ECHO-815.
What changes
Replace the substring regex used in all log panels:
with level-anchored patterns:
\b(ERROR|FATAL|CRITICAL|Exception|Traceback)\b|\[(error|fatal|critical)\]\b(WARN|WARNING)\b|\[(warn|warning)\]Touches 14 places in
configmap-grafana-dashboards.yaml:logfiltervariable values + 1 query stringWhy this drops the noise
_is a word character in RE2.\berror\bdoes NOT match insidetranscript_error/latest_error/latest_error_code— directus's loudest column-name offenders disappear without exclusion lists.ERROR|FATAL|CRITICALis case-sensitive on purpose: matches pythonlogginglevel prefixes (confirmed: echo server uses standardlogging.getLogger(...)).Exception/Tracebackcovers tracebacks.\[(error|fatal|critical)\]covers nginx-style bracketed lowercase.What this drops that you might miss
INFOline that talks about an error). Mostly low-signal.The real fix (
promtailJSON pipeline → extractlevelas a label → panels filter on{level=~"error|fatal|critical"}) supersedes this. Filed in ECHO-815 as a follow-up.Verification
Locally JSON-parsed each changed
exprstring and ran the inner regex against sample log lines:ERROR: kafka conn refusedException in thread mainnginx [error] 502 bad gatewayINFO: latest_error column was nullINFO: transcript_error field updatedINFO accessing /api/transcript_errors?id=42WARNING: slow queryHelm not installed locally so couldn't render the full template — the changes are pure data inside a block scalar, no template substitution touched.
Confidence
Medium-high. The change is mechanical and the regex was verified against the directus-column false-positives. The unknown is panel-level UX: warning panels now only show WARN/WARNING (instead of also catching the bottom of the error spectrum). If that turns out to be wrong, easy follow-up.
Refs: ECHO-815