[BUG] Temporarily fix issue #1586 by marking some failed tests as non-strict expected fail.

[EmanAbdelhaleem]

Metadata

• Reference Issue: Temporarily fix issue [BUG] resolve failing and flaky tests in CI, and race conditions in tests using server #1586

Details

• Running the pytest locally, I found only one failed test which is: tests/test_runs/test_run_functions.py::test__run_task_get_arffcontent_2
• However, when trying to go through the failed tests in the recent runed jobs in different recent PRs, I found many other failed tests, I picked some of them and tried to make a kind of analysis, and here are my findings:

Primary Failure Patterns

1. OpenML Test Server Issues (Most Common)
   The majority of failures are caused by:

• OpenMLServerError: Unexpected server error when calling https://test.openml.org/... with Status code: 500
• Database connection errors: Database connection error. Usually due to high server load. Please wait N seconds and try again.
• Timeout errors: TIMEOUT: Failed to fetch uploaded dataset

2. Cache/Filesystem Issues

• ValueError: Cannot remove faulty tasks cache directory ... Please do this manually!
• FileNotFoundError: No such file or directory

3. Data Format Issues

• KeyError: ['type'] not found in axis
• KeyError: ['class'] not found in axis
• KeyError: ['Class'] not found in axis

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.70%. Comparing base (bd8ae77) to head (ca65e9d).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1587       +/-   ##
===========================================
- Coverage   79.14%   52.70%   -26.44%     
===========================================
  Files          36       36               
  Lines        4320     4320               
===========================================
- Hits         3419     2277     -1142     
- Misses        901     2043     +1142     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.