Skip to content

Prevent traffic before storage initialization#5669

Draft
mikaelweave wants to merge 17 commits into
mainfrom
mikaelweave-investigate-work-item-197936
Draft

Prevent traffic before storage initialization#5669
mikaelweave wants to merge 17 commits into
mainfrom
mikaelweave-investigate-work-item-197936

Conversation

@mikaelweave

@mikaelweave mikaelweave commented Jul 11, 2026 •
edited by azure-boards Bot
Loading

Copy link
Copy Markdown
Contributor

Description

Returns an unhealthy health check while storage initialization is incomplete, preventing Kubernetes from routing traffic before search parameters and the SQL FHIR model are ready.

Related issues

AB#197936

Testing

  • dotnet test src\Microsoft.Health.Fhir.Api.UnitTests\Microsoft.Health.Fhir.R4.Api.UnitTests.csproj --no-restore --filter FullyQualifiedName~StorageInitializedHealthCheckTests
  • E2E readiness validation is delegated to the PR pipeline.

FHIR Team Checklist

  • CI is green before merge

Semver Change

Patch

@mikaelweave
Prevent traffic before storage initialization
d30e4bf 
Return an unhealthy health check while storage initialization is incomplete so Kubernetes does not route requests to a pod whose search parameters and SQL model are not ready.

AB#197936

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
mikaelweave requested a review from a team as a code owner July 11, 2026 02:31
@mikaelweave mikaelweave added this to the FY27\Q1\2wk\2wk01 milestone Jul 11, 2026
@mikaelweave mikaelweave added Area-API Area related to REST APIs. Azure Healthcare APIs Label denotes that the issue or PR is relevant to the FHIR service in the Azure Healthcare APIs Bug Bug bug bug. Bug-Reliability Reliability related bugs. No-ADR ADR not needed No-PaaS-breaking-change labels Jul 11, 2026
mikaelweave
mikaelweave commented Jul 11, 2026
View reviewed changes
Comment thread src/Microsoft.Health.Fhir.Api/Features/Health/StorageInitializedHealthCheck.cs
TimeSpan waited = Clock.UtcNow - _started;
if (waited < TimeSpan.FromMinutes(5))
{
return Task.FromResult(new HealthCheckResult(HealthStatus.Degraded, $"Storage is initializing. Waited: {(int)waited.TotalSeconds}s."));

@mikaelweave mikaelweave Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMPORTANT - we cannot return 200 from health check until the storage is initialized, else containers will have traffic routed and will return 503.

@mikaelweave
mikaelweave marked this pull request as draft July 11, 2026 02:51
@codecov-commenter

codecov-commenter commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.54%. Comparing base (b90efca) to head (4a84926).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5669      +/-   ##
==========================================
+ Coverage   77.52%   77.54%   +0.01%     
==========================================
  Files        1004     1006       +2     
  Lines       36924    36964      +40     
  Branches     5600     5605       +5     
==========================================
+ Hits        28627    28662      +35     
- Misses       6938     6946       +8     
+ Partials     1359     1356       -3

see 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mikaelweave and others added 16 commits July 10, 2026 22:02
@mikaelweave
Configure storage initialization health timing
4a84926 
AB#197936

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
docs: add separate health probes design spec
901b2a8 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
docs: amend health probes spec and add implementation plan
fbb5ed7 
Incorporate all 13 external-review decisions into the design spec and add the 11-task TDD implementation plan for splitting the FHIR health probes into startup/ready/live/check endpoints.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
feat(health): add HealthCheckTags constants
c8c952f 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
feat(health): add startup/ready/live route constants
71fe724 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
refactor(health): remove StartupDegradedDelay from storage-init config
161640b 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mikaelweave
refactor(health): make StorageInitializedHealthCheck a pure startup gate
8f3c533 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mikaelweave
feat(health): validate storage-init config on start
d6dfcf0 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
fix(health): make ImproperBehaviorHealthCheck state publication atomic
0dbc3d0 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
feat(health): tag health checks for startup/readiness probes
5e06ec7 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mikaelweave
feat(health): map startup/ready/live endpoints with tag predicates an…
f1bde3a 
…d startup assertion

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
refactor(health): share DataStoreHealthCheck name constant and clarif…
a1382c6 
…y startup assertion message

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
test(health): registration tag-resolution tests for readiness/startup
f9b9c4b 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
test(health): aggregate-status tests for four probe predicates (depen…
ef577a0 
…dency-free)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
fix(test): poll health/startup for storage-init failure E2E assertions
0f83a3d 
The pure-gate StorageInitializedHealthCheck is tagged probe:startup and is
excluded from /health/check, so the init-failure E2E tests must poll
/health/startup to observe the Unhealthy (503) startup-gate state they assert.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
@mikaelweave
refactor(health): centralize probe predicates so tests guard producti…
b63dd72 
…on routing

Extracts the four endpoint selection predicates from the inline UseFhirServer
lambdas into HealthProbePredicates. UseFhirServer and the startup registration
assertion now consume it, and the registration/endpoint unit tests reference the
same delegates instead of re-declaring copies. This closes a drift gap: editing
the real probe routing (e.g. dropping the /health/check startup exclusion or the
readiness OR-logic) now breaks the tests, guarding the AB#197936 fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ff23dc3f-596e-4d5b-b473-bc30ddb1684b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Area-API Area related to REST APIs. Azure Healthcare APIs Label denotes that the issue or PR is relevant to the FHIR service in the Azure Healthcare APIs Bug Bug bug bug. Bug-Reliability Reliability related bugs. No-ADR ADR not needed No-PaaS-breaking-change

Projects

None yet

Milestone

FY27\Q1\2wk\2wk01

Development

Successfully merging this pull request may close these issues.

2 participants

@mikaelweave @codecov-commenter