[BUG] Diagnose and fix current test failures in CI and local

[jgyasu]

Update - 29 Dec, 2025: Now the tests are failing in local as well as on CI as pointed by @fkiraly

While following the testing-your-installation guide, running pytest locally results in multiple test failures.

We should investigate the failing tests and debug them.

Todos after resolving tests:

• remove the pytest.xfails
• set sklearn-only: "false" in the include jobs in the test.yml test job

Note: originally it was thought that only local tests fail while main CI passes. This was a confusion that was caused by a state of the repo in which main was no longer used and develop used as the main branch, and for releases. This has since been rectified (main is now the main branch), so remote CI and local failures agree.

[rohansen856]

I would like to work on this issue. Looking into it now.

[PGijsbers]

Quite a few tests rely on connections to the servers (both production and test). In some cases they even rely on server state. This means that it's possible for tests to fail "randomly". I'm on mobile so I can't easily search but I do believe there is already a different issue about making tests rely on connections less by a combination of reducing the surface each individual test tests, and using mocked responses.

[rohansen856]

Did some digging and tried to backtrack the test calls and found this:

Call Stack:

openml.tasks.get_task(115)
  → dataset.retrieve_class_labels(task.target_name)
    → self.features.values()
      → self._load_features()
        → _get_dataset_features_file()
          → _get_features_xml(dataset_id=20)
            → OpenMLServerException (code 274)

The 20th dataset throws error when called from local: https://test.openml.org/api/v1/xml/data/features/20

Got this when i checked the url myself:

Although i am still looking as to the root cause of the failures i have these doubts about why it locally fails but CI Passes:**

• CI configuration in /.github/workflows/test.yml runs tests in two separate steps:
  1. Test server tests (line 106-113): Excludes production marker ->marks='sklearn and not production'
  2. Production server tests (line 114-121): Only runs production marker ->marks='sklearn and production'
• These failing tests have @pytest.mark.sklearn() but NOT @pytest.mark.production()

Will update on the matter as i dig more into it!

[fkiraly]

@EmanAbdelhaleem agreed to add temporary skips in a PR so other PR do not get blocked (or delayed) by the recurring failures. @rohansen856 will investigate and try to debug.

[EmanAbdelhaleem]

I am working on marking these failures as xfail temporarily

[fkiraly]

@jgyasu, we need to investigate exactly which tests are failing when:

• remote or local, how many and which tests are failing?
• sklearn-only False vs True flag in CI

[fkiraly]

Quite a few tests rely on connections to the servers (both production and test). In some cases they even rely on server state. This means that it's possible for tests to fail "randomly". I'm on mobile so I can't easily search but I do believe there is already a different issue about making tests rely on connections less by a combination of reducing the surface each individual test tests, and using mocked responses.

@PGijsbers, the test failures are not random but consistent, always the same on all the CI steps.

[fkiraly]

Failures appear under the following conditions on CI:

• always: 41 failures
• sklearn-only is False - 42 additional failures
• windows operating system, windows-latest - 41 failures

These should be investigated.