What are you really trying to do?
Ensure worker shutdown is safe when activity completions are still in flight.
Describe the bug
If an activity completes while TemporalWorker is shutting down, the native worker may already have been finalized. Finalization removes the underlying Core worker, then the late completion reaches temporal_core_worker_complete_activity_task and unwraps the missing worker at worker.rs:852, aborting the .NET process:
called `Option::unwrap()` on a `None` value
fatal runtime error: failed to initiate panic, error 5, aborting
This timing-sensitive race was observed while running:
ExecuteWorkflowAsync_Cancel_ProperlyCanceledExecuteWorkflowAsync_StdlibSemaphore_NonAsyncDeadlocks
The runner named the cancellation test as active at the time of the crash, but both tests can leave activity completions in flight as worker shutdown begins.
Minimal Reproduction
Configure the test harness to use Temporal Cloud, then run either test:
dotnet test tests/Temporalio.Tests/Temporalio.Tests.csproj \
--filter "FullyQualifiedName~ExecuteWorkflowAsync_Cancel_ProperlyCanceled"
dotnet test tests/Temporalio.Tests/Temporalio.Tests.csproj \
--filter "FullyQualifiedName~ExecuteWorkflowAsync_StdlibSemaphore_NonAsyncDeadlocks"
The race may require repeated runs.
Environment/Versions
- OS and processor: GitHub Actions
ubuntu-latest, x64
- Temporal environment: ephemeral Temporal Cloud namespace using mTLS
- Temporal .NET SDK
1.18.0.0
- sdk-core
78ac17e4bdaffdb54eb1c925a07d1fd40ce7b7f6
- Failing CI job
Additional context
PR #823 marks both tests as NeedsCloudAdaptation. A durable fix should either wait for in-flight activity completions before native worker finalization or safely reject completions submitted after finalization.
What are you really trying to do?
Ensure worker shutdown is safe when activity completions are still in flight.
Describe the bug
If an activity completes while
TemporalWorkeris shutting down, the native worker may already have been finalized. Finalization removes the underlying Core worker, then the late completion reachestemporal_core_worker_complete_activity_taskand unwraps the missing worker atworker.rs:852, aborting the .NET process:This timing-sensitive race was observed while running:
ExecuteWorkflowAsync_Cancel_ProperlyCanceledExecuteWorkflowAsync_StdlibSemaphore_NonAsyncDeadlocksThe runner named the cancellation test as active at the time of the crash, but both tests can leave activity completions in flight as worker shutdown begins.
Minimal Reproduction
Configure the test harness to use Temporal Cloud, then run either test:
The race may require repeated runs.
Environment/Versions
ubuntu-latest, x641.18.0.078ac17e4bdaffdb54eb1c925a07d1fd40ce7b7f6Additional context
PR #823 marks both tests as
NeedsCloudAdaptation. A durable fix should either wait for in-flight activity completions before native worker finalization or safely reject completions submitted after finalization.