fix collector error handling - #1032
Draft
kaysond wants to merge 3 commits into
Draft
Conversation
…dling # Conflicts: # collector/pkg/collector/metrics.go # webapp/backend/pkg/models/collector/smart.go
smartctl.exit_status becomes a typed bitmask matching smartctl.h. Bits 0-2 (FAILCMD, FAILDEV, FAILSMART) mean smartctl could not produce usable output, so the collector stops publishing those results and the backend rejects them before writing to either database. The remaining bits describe problems with the disk itself, which is data worth keeping. Exit status 2 is ambiguous: it also means the drive was in a low power mode and `-n` asked smartctl to exit. When that bit is the only one set and the configured smart args contain -n/--nocheck, the drive is simply asleep, so it is skipped without a notification. Adds POST /api/collector/error so the collector can report scan failures, per device `smartctl --info` failures (previously swallowed, which made the device silently disappear) and fatal `smartctl --xall` failures. The backend turns those into notifications, controlled by a new opt-out metrics.notify_collector_errors setting. LogSmartctlExitCode was an if/else if chain, so only the lowest set bit was ever logged. It now reports every set bit. Closes #944 Closes #945
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #944
Closes #945
Builds on the existing
wipcommit —masteris merged in and the work is finished in commits ontop, so the diff of what changed since
wipstays reviewable.#944 — stop storing smartctl's failure output
smartctl.exit_statusis now a typed bitmask(
webapp/backend/pkg/models/collector/smartctl_exit_status.go) matchingsmartctl.h,
used by both
collector.SmartInfoandmodels.Scan.The bits split cleanly in two:
FAILCMD,FAILDEV,FAILSMART) mean smartctl could not produce usable output —SmartctlExitStatus.IsFatal()Two independent guards, because either side can be older than the other:
UploadDeviceMetricsrejects such a payload with 400 before touching either databaseLogSmartctlExitCodealso becameLogSmartctlExitStatusand now logs every set bit. The oldversion was an
if / else ifchain, so exit status0x18only ever logged "failing disk" andswallowed the pre-fail bit.
#942 (
-n standby)Exit status 2 is shared: it means both "device open failed" and "device is in a low power mode and
-nasked me to exit". WhenFAILDEVis the only bit set and the configuredmetrics_smart_argscontain-n/--nocheck, the drive is simply asleep — the collector logs thatat info level and moves on without publishing and without notifying.
That is deliberately conservative: a drive that has genuinely gone away also disappears from
smartctl --scan, so it never reachesCollect()in the first place.#945 — notifications for collector failures
New route
POST /api/collector/error, handled byhandler.CollectorError. The collector reports:smartctl --scanfailures (no device attached to the report)smartctl --infofailures — previously these were swallowed with//ignore errors.and the device silently vanished from the dashboard, because a device that fails
--infonevergets a ScrutinyUUID
smartctl --xallfailures that are fatal per the aboveDetect.Start()is split intoScan()andInfo(), andInfo()returns a[]detect.DeviceInfoErrorso each failing device is reported with its own error rather than oneshared message.
New setting
metrics.notify_collector_errors, defaulting to true (opt-out, as the issue asked),with a migration (
m20260725120000) and a "Notify - Collector Errors" control in the settings dialog.Differences from the
wipcommitPOST /api/device/:scrutiny_uuid/smarthad been dropped fromserver.go— restored. Withoutit the collector cannot upload anything.
lo.Filterpredicate kept devices whereScrutinyUUID.IsNil()was true, i.e. exactly thedevices that were meant to be dropped. Resolved in favour of the loop that landed on
masterinfix(collector): keep only devices with non-nil ScrutinyUUID #991.
/api/collector_scan_errorand/api/device/:scrutiny_uuid/collector_error)collapsed into one. The device-scoped one had to invent a UUID via
GenerateScrutinyUUID("", "", deviceName)for the--infofailure case, which produces an ID thatmatches no real device; the single endpoint carries the device name and an optional UUID instead.
SmartctlExitStatuslives inwebapp/backend/pkg/models/collectorrather thancollector/pkg/models, so the dependency keeps pointing collector → webapp models (the directiondetect.goalready uses) instead of adding the reverse edge.HasFailCmd()/HasFailDev()/… replaced byHas(flags),IsFatal()andDescriptions().Tests
smartctl_exit_status_test.go— which exit codes are fatal, and that all set bits get reportedTestHasPowerModeCheck— the-n/--nocheckforms smartctl acceptsTestUploadDeviceMetricsRoute_RejectsFatalSmartctlExitStatus— uploads good data, then the samepayload with exit status 2, and asserts 400 plus that the stored summary is byte-for-byte unchanged
TestCollectorErrorRoute— posts a collector error and asserts a notification actually reaches awebhook, with the right failure type and message
AI disclosure
Per AI_POLICY.md: the commits after
wipwere written by Claude Code (Opus 5).It did the merge conflict resolution, the implementation, and the checks below, all inside containers
(
golang:1.25-trixiewith aninfluxdb:2.8sidecar,node:24-trixiewith headless Chromium);nothing was installed on the host.
Automated verification performed:
go build ./...andgo vetclean; cross-compiled forwindows/amd64,darwin/arm64andfreebsd/amd64so the per-OSdetectfiles all still compilego test ./...against a live influxdb — all packages pass, including thewebapp/backend/pkg/webintegration suitenpm run build:prodandnpx ng test --watch=false --browsers=ChromeHeadless --code-coveragegofmtcleanThat is the extent of the verification behind this PR as opened. It has not been run against real
hardware, and in particular the
-n standbypath has not been exercised against a drive that isactually asleep.