Getting `Attempt to execute managed code after the .NET runtime thread state has been destroyed.` after WPF window close and thread exit

[cyanite]

Description

After updating our WPF application to .NET 10, we see a new periodic Fail_Fast failure when closing windows.

For some of our windows, we use a spearate STA thread to run the WPF dispatcher on, so we start the thread, set up and open the window, and then quit the thread after the window is closed.

Sometimes, during termination of those STA threads, msctf fires a COM callback during FLS cleanup, or something. Eventually coreclr enters OsAttachThread and fails with:

Attempt to execute managed code after the .NET runtime thread state has been destroyed.

This doesn't happen all the time, so it's hard to reproduce. According to our customer, it mainly happens when opening/closing these windows shortly after application start. It doesn't happen for any particular person often, but since they have many people, it happens several times daily for someone.

The code on our end is essentially unchanged for many years, since .NET Framework times, and has worked correctly on .NET 5, 6 and 8.

The stack trace from the memory dump:

 # Child-SP          RetAddr               Call Site
00 000000af`b290ef90 00007ffa`3e26dd00     KERNELBASE!RaiseFailFastException+0x188
01 000000af`b290f570 00007ffa`3e23835b     coreclr!__FreeBuildAssertFail+0xf8 [D:\a\_work\1\s\src\runtime\src\coreclr\utilcode\debug.cpp @ 498] 
02 (Inline Function) --------`--------     coreclr!OsAttachThread+0xd14ac [D:\a\_work\1\s\src\runtime\src\coreclr\vm\ceemain.cpp @ 1721] 
03 000000af`b290f610 00007ffa`3e166e5b     coreclr!EnsureTlsDestructionMonitor+0xd14c3 [D:\a\_work\1\s\src\runtime\src\coreclr\vm\ceemain.cpp @ 1764] 
04 000000af`b290f640 00007ffa`3e09376e     coreclr!SetThread+0x53 [D:\a\_work\1\s\src\runtime\src\coreclr\vm\threads.cpp @ 376] 
05 000000af`b290f670 00007ffa`3e09359f     coreclr!SetupThread+0x186 [D:\a\_work\1\s\src\runtime\src\coreclr\vm\threads.cpp @ 743] 
06 000000af`b290f720 00007ffa`3e3a4272     coreclr!SetupThreadNoThrow+0x5f [D:\a\_work\1\s\src\runtime\src\coreclr\vm\threads.cpp @ 822] 
07 000000af`b290f7b0 00007ffa`3e1e8d1b     coreclr!COMToCLRWorker+0x52 [D:\a\_work\1\s\src\runtime\src\coreclr\vm\comtoclrcall.cpp @ 446] 
08 000000af`b290f820 00007ffb`0115dfc6     coreclr!GenericComCallStub+0x4b [D:\a\_work\1\s\src\runtime\src\coreclr\vm\amd64\GenericComCallStubs.asm @ 111] 
09 000000af`b290f8a0 00007ffb`0115abac     msctf!CThreadInputMgr::_OnThreadFocus+0x206
0a 000000af`b290f910 00007ffb`0113e46d     msctf!CThreadInputMgr::Suspend+0x104
0b 000000af`b290f940 00007ffb`0113dfc2     msctf!CThreadInputMgr::OnActivationChange+0xb1
0c 000000af`b290f9c0 00007ffb`011a88b8     msctf!CThreadInputMgr::Deactivate+0x72
0d 000000af`b290fa60 00007ffb`011a8750     msctf!CThreadInputMgr::~CThreadInputMgr+0x144
0e 000000af`b290fa90 00007ffb`0116a210     msctf!CThreadInputMgr::`vector deleting destructor'+0x14
0f 000000af`b290fac0 00007ffb`01170868     msctf!CThreadInputMgr::Release+0x70
10 000000af`b290faf0 00007ffb`01172e47     msctf!Microsoft::WRL::ComPtr<IConnectionMonitor>::InternalRelease+0x20
11 000000af`b290fb20 00007ffb`01170897     msctf!CTextInputClientWrapper::CInputLanguageEventMonitor::Uninitialize+0x23
12 000000af`b290fb50 00007ffb`0115b55e     msctf!CTextInputClientWrapper::_Uninitialize+0x1f
13 000000af`b290fb80 00007ffb`0115b161     msctf!FreeSYSTHREAD+0x11e
14 000000af`b290fc00 00007ffb`0115b062     msctf!TF_UninitThreadSystem+0x49
15 000000af`b290fc30 00007ffb`01d2736f     msctf!CicFlsCallback+0x42
16 000000af`b290fc90 00007ffb`01d68250     ntdll!RtlpFlsDataCleanup+0xff
17 000000af`b290fcf0 00007ffb`01d67ca6     ntdll!LdrShutdownThread+0x2d0
18 000000af`b290fdf0 00007ffb`01a2e960     ntdll!RtlExitUserThread+0x46
19 000000af`b290fe30 00007ffb`01d67c1c     kernel32!BaseThreadInitThunk+0x20
1a 000000af`b290fe60 00000000`00000000     ntdll!RtlUserThreadStart+0x2c

The assert we run into was introduced in dotnet/runtime#112809 and later, after a bug report involving c++ and IJW, was refined in dotnet/runtime#118827

Some previous bug reports involving this assert are dotnet/runtime#117538 and dotnet/runtime#118741. The last one is still open as of this writing.

Reproduction Steps

We open our application (a pension policy management system), open a policy (which will start a thread and open the window with it), close the window again.

Expected behavior

That it doesn't fail, which has been the case until now.

Actual behavior

Infrequently, the entire application shuts down (due to the Fail_Fast).

Regression?

This worked before .NET 10, probably before the mentioned PR that introduced the assert, although it hasn't been precisely bisected.

Known Workarounds

We currently don't know what to do to work around this, and that's a problem for our product.

Configuration

Information from windbg:

    Information from resource tables:
        CompanyName:      Microsoft Corporation
        ProductName:      Microsoft® .NET
        InternalName:     CoreCLR.dll
        OriginalFilename: CoreCLR.dll
        ProductVersion:   10,0,926,27113 @Commit: 901ca941248413c79832d2fdbd709da0c4386353
        FileVersion:      10,0,926,27113 @Commit: 901ca941248413c79832d2fdbd709da0c4386353
        FileDescription:  .NET Runtime
        LegalCopyright:   © Microsoft Corporation. All rights reserved.
        Comments:         Flavor=Retail

The operating system is Windows 11.

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[MichalPetryka]

This worked before .NET 10, probably before the mentioned PR that introduced the assert, although it hasn't been precisely bisected.

Code doing such behaviour was always invalid and would corrupt the process state, the changes made in runtime only made it error out more eagerly.

Are you doing any OS interop related to the stacktrace here? If not then it's likely a bug in WPF doing so and should probably be transferred there.

[cyanite]

Code doing such behaviour was always invalid and would corrupt the process state, the changes made in runtime only made it error out more eagerly.

Right, although the work-around for IJW seems to just let it pass? We haven't observed any problems in the past, anyway, and the code has worked like this for 15 years.

Are you doing any OS interop related to the stacktrace here? If not then it's likely a bug in WPF doing so and should probably be transferred there.

I don't think we are doing any interop. Certainly everything in this stack trace is unrelated to anything we are doing. I didn't realize WPF was a different project, sorry.

[MichalPetryka]

the work-around for IJW seems to just let it pass

AFAIR it was a temporary "let this happen for now, it was always broken and we can't get it fixed quickly" see the description: This replaces deterministic fails with intermittent hangs and crashes for backward compatibility., cc @jkotas on that.

I didn't realize WPF was a different project, sorry.

It's handled by a separate team under dotnet/wpf

[cyanite]

It's handled by a separate team under dotnet/wpf

Right, so it seems there are some possibilities for what's going on:

1. It's a bug in WPF that has been there for a while, but only surfaced once this assert was broadened (customers might have observed weird behavior, but not reported it to us).
2. It's a newer bug in WPF, between the .NET 8 to 10 move.
3. It's a bug in our code. I realize it's not completely typical to make a new thread with a dispatcher for each window, but this is ancient code that has run like this since WPF was invented.

Should I post a separate bug to the WPF project and link to this, or what should the process be?

[MichalPetryka]

I assume it's probably 1 and a report to WPF that links here is the correct thing to do but up to @jkotas to confirm.

[jkotas]

Yes, this looks like an issue with WPF. WPF should unregister the managed ITfThreadMgr COM object so that it is not called late during thread shutdown.

I expect that the crash is not going to reproduce in .NET 11 since #11575 switched WPF to use regular managed C++ that will suppress the assert (dotnet/runtime#118827).