slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
3033720	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1530232	dotnet/maui	Controls SoftInput,Stepper,Switch,SwipeView	Log
3033692	dotnet-sdk	Mac arm64 AOT	Log
1530244	dotnet/runtime	osx-x64 Release Libraries_CheckedCoreCLR	Log
3033671	dotnet-sdk	TestBuild: macOS (x64)	Log	#63320
3033606	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
1530163	dotnet/aspnetcore	Tests: macOS	Log
3033590	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1530106	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131482
1530045	dotnet/maui	Controls Shell	Log
3033587	dotnet-maui	Controls Shell	Log
3033538	dotnet-maui	RunOniOS_MauiNativeAOT ARM64
3033523	dotnet-maui	RunOniOS_MauiDebug_CoreCLR ARM64	Log
3033520	dotnet-maui	Build macOS	Log
1529949	dotnet/aspnetcore	Tests: macOS	Log
3033481	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1529856	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36838
1529859	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131471
3033452	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
3033438	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
1529793	dotnet/maui	Controls Shell	Log
3033434	dotnet-maui	RunOniOS_BlazorDebug x64	Log
1529749	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
3033389	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
3033399	dotnet-android	macOS > Tests > MSBuild+Emulator 1	Log
1529685	dotnet/maui	Controls Shell	Log	dotnet/maui#36657
1529684	dotnet/maui	Controls Shell	Log	dotnet/maui#36653
1529649	dotnet/maui	Controls Shell	Log
1529578	dotnet/aspnetcore	Tests: macOS	Log
3033360	dotnet-performance	Performance maui_scenarios_android_innerloop Default CoreCLR_Debug_InnerLoop windows 22H2 x64 Pixel8 main	Log
1529562	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131455
1529485	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36750
1529460	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36676
1529453	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#131310
1529434	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#131309
1529387	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#130023
1529368	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1529412	dotnet/runtime	osx-x64 Debug PALTests	Log
1529315	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129683
1529295	dotnet/runtime	coreclr Pri0 Runtime Tests Run R2R_CG2_HotColdSplitting osx x64 checked	Log
1529291	dotnet/maui	Controls Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#36866
1529281	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#131027
1529278	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131027
1529267	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R osx x64 checked	Log
1529233	dotnet/runtime	iossimulator-x64 Release AllSubsets_NativeAOT	Log
3033042	dotnet-maui	RunOniOS_MauiDebug_CoreCLR ARM64	Log
1529141	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131431
3032968	dotnet-performance	Performance maui_scenarios_android Default CoreCLR_Debug windows 22H2 x64 Pixel8 main	Log
1528851	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log
3032770	dotnet-maui	RunOniOS_BlazorRelease x64	Log
3032771	dotnet-maui	Controls Shell	Log
1528761	dotnet/maui	Controls Shell	Log	dotnet/maui#36838
1528766	dotnet/maui	Controls CollectionView	Log
1528714	dotnet/maui	Controls Shell	Log
1528364	dotnet/aspnetcore	Tests: macOS	Log
1528219	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131375
1528129	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36328
3032233	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
1528104	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131110
1528082	dotnet/aspnetcore	Tests: macOS	Log
1528042	dotnet/maui	Controls Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#36109
1527989	dotnet/maui	Controls Shell	Log
1528004	dotnet/maui	Controls Layout	Log	dotnet/maui#36476
3032309	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
3032407	dotnet-performance	Performance powershell ubuntu 2204 arm64 Ampere main	Log
1528013	dotnet/aspnetcore	Tests: macOS	Log
1527889	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1527926	dotnet/maui	Controls Shell	Log	dotnet/maui#36676
1527892	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
3032257	dotnet-maui	Build macOS	Log
1527770	dotnet/maui	Controls Accessibility,ActionSheet,ActivityIndicator,Animation,AppLinks	Log	dotnet/maui#36741
3032221	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
3032231	dotnet-maui	Build macOS	Log
1527763	dotnet/aspnetcore	Tests: macOS	Log
1527719	dotnet/maui	Controls Shell	Log
1527671	dotnet/maui	Controls Shell	Log	dotnet/maui#36817
1527631	dotnet/maui	Controls Shell	Log	dotnet/maui#36476
3032133	dotnet-efcore	macOS	Log
1526425	dotnet/maui	Controls CollectionView	Log
1527549	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1640
1527525	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1639
1527477	dotnet/aspnetcore	Tests: macOS	Log
1527413	dotnet/aspnetcore	Tests: macOS	Log
3031965	dotnet-maui	Build macOS	Log
1527365	dotnet/maui	Controls Shell	Log	dotnet/maui#36654
1527366	dotnet/maui	Controls Shell	Log	dotnet/maui#36658
1527333	dotnet/aspnetcore	Tests: macOS	Log
1527318	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36784
1527179	dotnet/maui	Controls Shell	Log
3031776	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64
1527070	dotnet/maui	Controls Layout	Log
1526984	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36633
1526949	dotnet/aspnetcore	Tests: macOS	Log
1526948	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36653
1526936	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log
1526788	dotnet/maui	Controls Shell	Log	dotnet/maui#36653
3031489	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
3031482	dotnet-maui	RunOniOS_MauiDebug x64	Log
3031471	dotnet-maui	Build macOS	Log
1526664	dotnet/aspnetcore	Tests: macOS	Log
Displaying 100 of 1379 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
47	294	1379

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.