Improved _inc_exc_datasets() and paradigms handling in benchmark()

[toncho11]

A re-written _inc_exc_datasets() that fixes issues and provides much needed checks. Before it was possible not to recognize correctly if the input is string or object and thus process the input incorrectly. Fixes: #654. It avoids some confusion induced by the old version.

Below is the code I used for testing:

# -------------------------------
# 1. Pipelines and datasets
# -------------------------------
pipelines_test = [
    # test the new router classifier
    {
        "paradigms": ["P300","LeftRightImagery"],
        "pipeline": make_pipeline(
            Covariances("oas"),
            CustomCspTransformer2(mode="high_electrodes_count"),
            CustomCspTransformer2(mode="low_electrodes_count"),
            MDM()
        ),
        "name": "TSLR"
    }
]

# P300 databases
from moabb.datasets import (
    BI2013a,
    BNCI2014_008,
    BNCI2014_009,
    BNCI2015_003,
    EPFLP300,
    Lee2019_ERP,
    BI2014a,
    BI2014b,
    BI2015a,
    BI2015b,
)

# Motor imagery databases
from moabb.datasets import (
    BNCI2014_001,
    Zhou2016,
    BNCI2015_001,
    BNCI2014_002,
    BNCI2014_004,
    #BNCI2015_004, #not tested
    AlexMI,
    Weibo2014,
    Cho2017,
    GrosseWentrup2009,
    PhysionetMI,
    Shin2017A,
    Lee2019_MI, #new
    Schirrmeister2017 #new
)

# -------------------------------
# 2. Run the benchmark
# -------------------------------

# Ensure the results folder exists
results_path = "./results/"
os.makedirs(results_path, exist_ok=True)

results = benchmark(
    pipelines=pipelines_test,
    evaluations=["WithinSession"], #, "CrossSession"
    paradigms=["P300", "LeftRightImagery"],
    results=results_path,
    output="./benchmark_results/",
    #exclude_datasets=["Stieger2021","Liu2024"], #must be OK
    #include_datasets=["BNCI2014-001"], # #must be OK
    #exclude_datasets=[Zhou2016(), Weibo2014()], #must be OK
    #include_datasets=[Zhou2016(), Weibo2014()], #must be OK
    #exclude_datasets=["Stieger2021","fsdfsdfs"], # should  fail  
    #include_datasets=[PhysionetMI(), Shin2017A(), Lee2019_MI()], #must be OK
    #include_datasets=[PhysionetMI(), Shin2017A(), "BNCI2014-001"], #should fail
    #exclude_datasets=[PhysionetMI(), Shin2017A(), "BNCI2014-001"], #should fail
    #exclude_datasets=[EPFLP300()], 
    #include_datasets=[Lee2019_ERP(), BI2015b()], #should be OK
    #include_datasets=[Lee2019_ERP(), "fsdfsdfdwwww"], #should fail
    #include_datasets=["fsdfsdfdwwww"], #should fail
    exclude_datasets = None, include_datasets = None, # should be OK
    overwrite=True,
    n_jobs=3,
    plot=False,
    n_splits=5
)

[bruAristimunha]

hey @toncho11,

can you fix the tests please:

FAILED moabb/tests/test_benchmark.py::TestBenchmark::test_benchmark_strdataset - ValueError: Invalid dataset codes in include_datasets: ['FakeDataset-p300-10-2--60-60--120-120--target-nontarget--c3-cz-c4', 'FakeDataset-ssvep-10-2--60-60--120-120--13-15--c3-cz-c4', 'FakeDataset-cvep-10-2--60-60--120-120--10-00--c3-cz-c4']
FAILED moabb/tests/test_benchmark.py::TestBenchmark::test_benchmark_objdataset - ValueError: Some datasets in include_datasets are not part of available datasets for the paradigms you requested in benchmark(): ['FakeDataset-p300-10-2--60-60--120-120--target-nontarget--c3-cz-c4', 'FakeDataset-ssvep-10-2--60-60--120-120--13-15--c3-cz-c4', 'FakeDataset-cvep-10-2--60-60--120-120--10-00--c3-cz-c4']
FAILED moabb/tests/test_benchmark.py::TestBenchmark::test_include_exclude - ValueError: Cannot specify both include_datasets and exclude_datasets.
===== 3 failed, 301 passed, 90 skipped, 207 war

[toncho11]

I need some more time on the code.