[kinyoklion]

Summary

Four server SDK unit tests fail intermittently on the macOS runner: three FileDataSourceTest retry tests, and FDv2DataSourceTest.FallbackAndRecoveryTasksWellBehaved, which has also failed on main itself (run 37331939401). One of those failures was not a test problem: it was the file data source dropping its retries after a read error. This change fixes that in the SDK, and fixes the tests.

A read error ends the file data source's retries

Cause. With auto-update on, the file data source retries a load that fails, so that a file read while it is being written is loaded once the write is complete (FileDataSource.cs). A load that is not a retry (Start, or a file-change notification) starts a new failure episode and clears the failure counts. When such a load failed to read a file, it logged an error and returned without scheduling a retry. A retry that was already pending then found no failures and did nothing. So the data stayed stale, or the source stayed uninitialized, until the next detected change, although the code's own comment on read errors says "another file-change notification is not guaranteed". Only a read error on a retry was retried.

The default reader already retries sharing violations inside the read for about 30 seconds, so an ordinary in-place write on Windows does not get here. A missing file (with missing files not skipped), an access error while a file is replaced, a sharing violation that outlasts the reader's own retries, and any error from a custom reader (FileDataSourceBuilder.FileReader, which is public) do.

Fix. With auto-update on, a read error on any load is charged to the path's retry budget, like a parse failure, and schedules a retry: up to five attempts, 600 ms apart, with a warning for each retry and an error when it gives up. With auto-update off, files are loaded once, and a read error is logged as before. The AutoUpdate documentation now says the source retries until the file "can be read and parsed".

This changes what an application sees when a file stays unreadable with auto-update on: a warning per retry and an error after five attempts, instead of one error, before the source waits for the next detected change.

File data source retry tests

Cause. The retry tests read through a ScriptedFileReader and count its reads, but they configured a real TempFile with AutoUpdate(true), so a live FileSystemWatcher watched the file's directory. On macOS the watcher has delivered a change notification for the test's own newly created file, 1 to 155 ms after Start(), although the test never changes the file. Every failing run logs detected file modification, reloading at that point, and the extra load explains each failure:

• ParseRetryStopsAfterMaxAttemptsAndDoesNotInit ("Expected: 5, Actual: 6") and PendingParseRetryIsSkippedIfAnExternalReloadAlreadySucceeded ("Expected: 2, Actual: 3"): the notification's load is one more read.
• TransientReadErrorDuringRetryDoesNotEndTheEpisode ("expected an event but did not get one"): the notification's load ran while the scripted reader threw, and hit the SDK defect above.

On Linux, a scratch copy of each test that makes one in-place write to its temp file at that point fails the same way.

Fix.

• The tests that count reads, and the ones that check that a retry, not a notification, applies the data, use a path in a directory that does not exist, as the multi-path tests already do. The watcher fails to construct for it, so the only loads are the ones the test starts and the retries they schedule. These tests read through the scripted reader already; the temporary file was never read.
• ParseRetryBudgetResetsForANewFailureEpisode needs a real change notification to start a new episode, so it keeps a watched file and changes it with an in-place write. It no longer asserts the read count after the first episode: it waits until the reads stop. A rewrite in place can produce more than one notification, and a later one could load the data that the test makes valid. So the test waits until no notification has arrived for 300 ms before it makes the content valid, and only the new episode's retry can then apply it.
• New tests pin the SDK fix: a read error on Start is retried; a read error on a load from outside the chain does not end a pending retry; read errors stop after five attempts; with auto-update off they are not retried; and a load from outside the chain gives a failing episode a fresh budget, which no test covered before (the existing test starts its second episode only after the first gave up, which clears the counts anyway). TransientReadErrorDuringRetryDoesNotEndTheEpisode now checks that the retry applies the whole file.

FDv2 fallback and recovery test

Cause. The applier runs its fallback and recovery actions (DisposeCurrent, then GoToNext or GoToFirst, then StartCurrent) on its own task after a 20 ms timeout. The test asserted after a fixed delay of 70 ms, and its mock recorded calls in plain fields that another thread wrote. The failing runs logged the recovery message and saw DisposeCurrent but not GoToFirst ("GoToFirst should be called on recovery"), so the assertion ran while the actions were still running.

Fix. The mock records its calls under a lock and reports them from the recorded list. The test waits until StartCurrent, the last action, has been recorded, then checks the whole call sequence. The negative check (no recovery when already at the first source) keeps its fixed wait.

Testing

• Each test this change touches fails against a mutant of the behavior it covers, one mutant at a time, with a clean build each time. The last seven rows were measured before the SDK fix was added:

Mutant	Fails
the SDK as on main	ReadErrorOnALoadThatIsNotARetryIsRetried, ReadErrorOnALoadThatIsNotARetryDoesNotEndThePendingRetries, ReadErrorRetryStopsAfterMaxAttemptsAndDoesNotInit
a load from outside the chain does not reset the budget	LoadThatIsNotARetryGivesAFailingEpisodeAFreshBudget
read errors are retried without a bound	ReadErrorRetryStopsAfterMaxAttemptsAndDoesNotInit
a read error goes on to apply what was read so far	7 tests, including TransientReadErrorDuringRetryDoesNotEndTheEpisode
read errors are retried with auto-update off	ReadErrorIsNotRetriedIfAutoUpdateIsOff
no retries are scheduled	ParseFailureFromPartialReadIsRetriedUntilContentIsComplete, ParseRetryAppliesWhenAlternateParserIsConfigured, ParseRetryStopsAfterMaxAttemptsAndDoesNotInit, TransientReadErrorDuringRetryDoesNotEndTheEpisode, ParseRetryBudgetResetsForANewFailureEpisode, and two multi-path tests
the retry budget allows one more attempt	ParseRetryStopsAfterMaxAttemptsAndDoesNotInit and a multi-path test
a read error on a retry ends the episode	TransientReadErrorDuringRetryDoesNotEndTheEpisode
a retry runs after Dispose	PendingParseRetryIsCanceledByDispose
a retry reloads after an external load already succeeded	PendingParseRetryIsSkippedIfAnExternalReloadAlreadySucceeded
retries stop for good once a path gives up	ParseRetryBudgetResetsForANewFailureEpisode and a multi-path test
recovery goes to the next source instead of the first	FallbackAndRecoveryTasksWellBehaved

• With a 150 ms pause between DisposeCurrent and GoToFirst in the applier, the old FDv2 test fails every time with the CI message, and the new one passes. CPU load alone does not tell them apart: both passed hundreds of runs with the machine's cores oversubscribed.
• With a change notification injected 5 or 150 ms after the watcher starts, the old file tests fail exactly the ones that flaked in CI, and the new ones pass.
• FileDataSourceTest and FDv2DataSourceTest (53 tests) passed 10 runs in a row on a loaded machine. The full server SDK test suite passes on net8.0 (1658 tests), and the SDK builds for netstandard2.0, net462 and net8.0.

_{Opened from termag}

SDK-3248

---

Note

Overview
File data source: With AutoUpdate(true), read failures on any load (Start, file-change reload, or retry) now use the same bounded retry path as parse failures (HandleReadFailure), instead of logging once and bailing on non-retry loads—which could leave data stale when no further watcher event arrives. With auto-update off, behavior stays single-load + error log. Public docs now say retries continue until the file can be read and parsed.

Tests: Retry tests that count scripted reads use an unwatched path so macOS FileSystemWatcher noise does not add extra loads; episode-reset tests wait for reads/notifications to settle. New coverage pins read-error retry, budget reset on external reload, and max-attempt give-up. FDv2 fallback/recovery test waits for the applier’s async actions via a locked call-sequence mock instead of a fixed delay.

^{Reviewed by Cursor Bugbot for commit 7e4e268. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7e4e268. Configure here.}