Some of the tests defined in the `BenchmarkDotNet.IntegrationTests.ManualRunning` project failed to run

[filzrev]

BenchmarkDotNet.IntegrationTests.ManualRunning test projects seems not be executed on CI.
And it need to manually invoke these tests.

But when running test on my environment (with VS Test Explorer).
Some tests are failed.

1. LocalNativeAotToolchainTests

Throws following exception.

System.IO.DirectoryNotFoundException : D:\projects\runtime\artifacts\packages\Release\Shipping provided as ilcPackages does NOT exist

It seems to be caused because ilcPath path is hardcoded.

2. MSBuildArgumentTests

Throws following exception.

BenchmarkDotNet.IntegrationTests.Xunit.MisconfiguredEnvironmentException : Failed to build benchmarks because the platform linker not found

It seems not support running tests from VS TestExplorer.
And need to set VC++ tools paths.

3. EmptyBenchmarksReportZeroTimeAndAllocated_InProcess
4. EmptyBenchmarksReportZeroTimeAndAllocated_Core
5. DifferentSizedStructsBenchmarksReportsNonZeroTimeAndZeroAllocated_InProcess

Some tests assertions fails with following message.

Actual time was not 0

[timcassell]

1. LocalNativeAotToolchainTests

Throws following exception.

System.IO.DirectoryNotFoundException : D:\projects\runtime\artifacts\packages\Release\Shipping provided as ilcPackages does NOT exist

It seems to be caused because ilcPath path is hardcoded.

It's very old, I think the test can just be deleted. NativeAot is included with dotnet sdk now, so we don't need to manually install it anymore. Also, we have BenchmarkDotNet.IntegrationTests.NativeAotTests which run automatically, so the manual test is obsolete.

2. MSBuildArgumentTests

Throws following exception.

BenchmarkDotNet.IntegrationTests.Xunit.MisconfiguredEnvironmentException : Failed to build benchmarks because the platform linker not found

It seems not support running tests from VS TestExplorer. And need to set VC++ tools paths.

Works on my machine™. 🤷‍♂

3. EmptyBenchmarksReportZeroTimeAndAllocated_InProcess 4. EmptyBenchmarksReportZeroTimeAndAllocated_Core 5. DifferentSizedStructsBenchmarksReportsNonZeroTimeAndZeroAllocated_InProcess

Some tests assertions fails with following message.

Actual time was not 0

This is expected, and is why they are manual tests and not run in the CI. They are testing extremely sensitive measurements which any bit of noise in the system may cause it to be flaky, and we cannot even guarantee results across different cpu architectures and generations. The purpose of these tests is to carefully verify any changes to the engine and codegen (e.g. #2336).

[adamsitnik]

LocalNativeAotToolchainTests

To run this test you would need to build a local copy of the Native AOT compiler. I don't believe it's worth investing time in doing that.

[filzrev]

Thanks for confirmations.

2. MSBuildArgumentTests

This issue is resolved by re-install Visual Studio on my environment.

---

I've created this to execute ManualRunning tests on GitHub Actions with run-tests-selected workflow.

Because some tests needs test patterns. But it take times to execute on automated CI tests.
(e.g. Unit tests that added at PR #2730)

If these tests are expected to be failed by default.
I'll try to add custom xUnit traits to skip these tests on CI.

[timcassell]

If these tests are expected to be failed by default.

They are not expected to fail by default, they are expected to be flaky.

I'll try to add custom xUnit traits to skip these tests on CI.

They should not be ran in CI automatically, but it's okay to run them from manual triggers.