Extend with SVM probes (#12)

* Implement new probe polynomial kernel SVM.

This commit implements a new probe type, the polynomial kernel SVM.
The SVM allows for linear probing as well as non-linear probing with
complete control over the nonlinearity of the probe. The intention of
this probe is as an alternative to the single-layer perceptron, as well
as analyzes into how much nonlinearity is needed for a specific
downstream task.

The SVM is trained using Bayesian hyperparameter search, using
Scikit-Optimize for the optimization. Expect longer run times compared
to the perceptron.

This update also includes refactoring of the existing code to make
additional probe types simpler to implement.

New with this update is also the possibility to store trained models. Use the `store_models` argument in the config file to toggle model storing. In case of `store_models=True`, the entire dataset is used for training the probe, no splitting into training and validation sets. The models are stored with the results together with a self-contained run script to provide easy loading.

* Update github workflow

Github actions now only run on the original repository and not on forks

* Remove unnecessary files

* Update documentation

* Add config checks.

Merged linear_probe.py into evaluation.py.
Renamed everything relating to storing the trained probe from using model to usingprobe.
Bug fixes.

* Bug fix linear and svm.

* Bug fix svm

* Add coef0 to optimizeable parameters for SVM.

* Align svm probe saving with linear model

* Add hyperparameter tuning of epsilon in support vector regression

* Update documentation and citation

Author: rikardvinge

.github/workflows/docs.yaml

@@ -17,6 +17,8 @@ concurrency:
 jobs:
   build:
     runs-on: ubuntu-latest
+    # Do not try to build documentation from forks
+    if: github.event.repository.fork == false
     steps:
       - uses: actions/checkout@v4
 
@@ -40,6 +42,8 @@ jobs:
   deploy:
     needs: build
     runs-on: ubuntu-latest
+    # Do not try to deploy documentation from forks
+    if: github.event.repository.fork == false
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}

CITATION.cff

@@ -13,6 +13,6 @@ authors:
   - family-names: Schneider
     given-names: Jannik
 title: "NeuCo-Bench"
-version: 1.0
+version: 1.1
 date-released: 2025-05-12
-url: https://github.com/embed2scale/benchmark
+url: https://github.com/embed2scale/NeuCo-Bench

README.md

@@ -3,7 +3,7 @@
 [![Docs](https://img.shields.io/badge/docs-MkDocs-526CFE?logo=materialformkdocs&logoColor=fff)](https://embed2scale.github.io/NeuCo-Bench/)  
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
-**TL;DR**: *Originally developed to evaluate challenge submissions for the 2025 EARTHVISION Challenge at CVPR ([competition details](https://www.grss-ieee.org/events/earthvision-2025/?tab=challenge)), NeuCo-Bench is now released for local benchmarking and evaluation - additional tech details in [http://arxiv.org/html/2510.17914](http://arxiv.org/html/2510.17914).*
+**TL;DR**: *Originally developed to evaluate challenge submissions for the 2025 EARTHVISION Challenge at CVPR ([competition details](https://www.grss-ieee.org/events/earthvision-2025/?tab=challenge)), NeuCo-Bench is now released for local benchmarking and evaluation - additional tech details in [the NeuCo-Bench paper](https://openaccess.thecvf.com/content/CVPR2026W/EarthVision/html/Vinge_NeuCo-Bench_A_Novel_Benchmark_Framework_for_Neural_Embeddings_in_Earth_CVPRW_2026_paper.html).*
 
 ---
 
@@ -156,13 +156,12 @@ For details on how to contribute, please see [CONTRIBUTING.md](.github/CONTRIBUT
 ## How to cite
 
 ```BibTeX
-@article{Vinge2025NeuCoBench,
-  author       = {Rikard Vinge and Isabelle Wittmann and Jannik Schneider and Michael Marszalek and Luis Gilch and Thomas Brunschwiler and Conrad M Albrecht},
-  title        = {NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation},
-  journal      = {arXiv preprint arXiv:2510.17914},
-  year         = {2025},
-  url          = {https://arxiv.org/abs/2510.17914},
-  doi          = {10.48550/arXiv.2510.17914},
-  note         = {Submitted on 19 Oct 2025},
+@InProceedings{Vinge_2026_CVPR,
+    author    = {Vinge, Rikard and Wittmann, Isabelle and Schneider, Jannik and Marszalek, Michael and Gilch, Luis and Brunschwiler, Thomas and Albrecht, Conrad M},
+    title     = {NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation},
+    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
+    month     = {June},
+    year      = {2026},
+    pages     = {8063-8074}
 }
 ```

benchmark/data/embeddings.py

@@ -1,9 +1,7 @@
 import json
 import logging
-import re
-import ast
 from pathlib import Path
-from typing import Optional, Set
+from typing import Optional, Set, Tuple
 
 import numpy as np
 import pandas as pd
@@ -40,9 +38,9 @@ def load_submission(
     file_path: Path,
     valid_ids: Set[str],
     expected_dim: int | None = None,
-    exclude_file: Optional[Path] = None,
+    exclude_file: Path | None = None,
     standardize: bool = True,
-) -> (pd.DataFrame, int):
+) -> Tuple[pd.DataFrame, int]:
     """
     Load and preprocess CSV of embeddings.
 

benchmark/data/labels.py

@@ -27,6 +27,7 @@ def get_annotations(folder_path: Union[str, Path]) -> pd.DataFrame:
 
     for csv_path in folder.glob("*.csv"):
         task_name, task_type = csv_path.stem.split("__", 1)
+        prev_len = len(entries)
         with csv_path.open(newline='') as csvfile:
             reader = csv.DictReader(csvfile)
             for row in reader:
@@ -40,7 +41,7 @@ def get_annotations(folder_path: Union[str, Path]) -> pd.DataFrame:
                     'task_name': task_name,
                     'task_type': task_type,
                 })
-        logger.info("Processed %s: %d valid entries", csv_path.name, len(entries))
+        logger.info("Processed %s: %d valid entries (%d total)", csv_path.name, len(entries) - prev_len, len(entries))
 
     if sorted_out:
         logger.warning("Skipped %d rows due to missing labels", sorted_out)

benchmark/evaluation/config.py

@@ -0,0 +1,43 @@
+
+from evaluation.probes.linear import validate_config as linear_validate_config
+from evaluation.probes.svm import validate_config as svm_validate_config
+
+def check_config(config: dict):
+    """Validates that the supplied configuration file is ok.
+    
+    Args:
+        Config (dict): Configuration dictionary
+    """
+
+    probe_info = {'linear': {'val_fun': linear_validate_config}, 
+                  'svm': {'val_fun': svm_validate_config}
+                  }
+
+    # Base parameters
+    for param in ['k_folds', 'probe_type', 'probe_params']:
+        assert param in config.keys(), f"Parameter `{param}` missing from  config."
+
+    # Optional parameters
+    if 'embedding_dim' in config.keys():
+        assert (config['embedding_dim'] is None) or (isinstance(config['embedding_dim'], int)), f"`embedding_dim` must be integer or None but got {type(config['embedding_dim'])}."
+
+    bool_params = ['standardize_embeddings', 'normalize_labels', 'enable_plots', 'update_leaderboard', 'output_fold_results', 'store_probes']
+    for p in bool_params:
+        if p in config.keys():
+            assert (isinstance(config[p], bool)), f"`{p}` must be boolean but got {type(config[p])}."
+    
+    if 'task_filter' in config.keys():
+        assert (config['task_filter'] is None) or (isinstance(config['task_filter'], list)), f"`task_filter` must be list or None but got {type(config['task_filter'])}."
+        if config['task_filter'] is not None:
+            for t in config['task_filter']:
+                assert isinstance(t, str), f"`task_filter elements must be strings but got {type(t)} for task {t} (task_filter: {config['task_filter']})."
+
+    # Probe parameters
+    if config['probe_type'] in probe_info.keys():
+        probe_info[config['probe_type']]['val_fun'](config['probe_params'])
+    else:
+        raise NotImplementedError(f"probe_type `{config['probe_type']}` is not implemented. Currently implemented probes are: {list(probe_info.keys())}")
+    
+    
+        
+

benchmark/evaluation/evaluation.py

@@ -1,20 +1,28 @@
-import argparse
 import json
 import logging
+import os
 from datetime import datetime
 from pathlib import Path
 import torch
+import numpy as np
+import pandas as pd
+from typing import Optional
+from sklearn.model_selection import ShuffleSplit
 
 from data.embeddings import load_submission
 from data.labels import get_annotations
-from evaluation.linear_probing import cross_validate
-from evaluation.results import save_results, summarize_runs
-from evaluation.utils import fix_all_seeds
+from evaluation.probes.linear import LinearFoldRunner
+from evaluation.probes.svm import SVMHPOptimizer
+from evaluation.results import save_results
+from evaluation.utils import fix_all_seeds, TaskResult
+from evaluation.visualisations import save_loss_curve
+
 
 # Logging setup
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger("benchmarking")
 
+
 def evaluate(submission_file: Path, 
              annotation_path: Path, 
              method_name: str, 
@@ -35,14 +43,6 @@ def evaluate(submission_file: Path,
         exclude_file: Path to file containing embedding IDs to exclude. If not provided, exclude no embeddings.
     """
     fix_all_seeds(seed=42)
-    device = torch.device(config.get("device", "cuda" if torch.cuda.is_available() else "cpu"))
-
-    # Determine device
-    device = (
-        torch.device("cuda" if torch.cuda.is_available() else "cpu")
-        if config.get("device", "auto") == "auto"
-        else torch.device(config["device"])
-    )
 
     # Load data
     annotation_df = get_annotations(annotation_path)
@@ -82,16 +82,15 @@ def evaluate(submission_file: Path,
             df=group,
             task_type=task_type,
             task_name=task_name,
-            device=device,
-            batch_size=config["batch_size"],
+            probe_type=config["probe_type"],
+            probe_params=config["probe_params"],
             n_splits=config["k_folds"],
-            epochs=config["epochs"],
             embedding_dim=embedding_dim,
-            learning_rate=config["learning_rate"],
             output_dir=run_dir,
             filename_prefix=submission_file.stem,
             enable_plots=config.get("enable_plots", True),
             output_fold_results=config.get("output_fold_results", False),
+            store_probes=config.get('store_probes', False),
         )
 
         task_q_scores[task_name] = result.q_statistic
@@ -100,6 +99,143 @@ def evaluate(submission_file: Path,
     save_results(experiment_name=experiment_name,
                  task_q_scores=task_q_scores,
                  task_acc_scores=task_acc_scores,
-                 output_dir=run_dir, config=config)
+                 output_dir=run_dir, 
+                 config=config,
+                 )
 
     logger.info("Finished evaluation.")
+
+
+def cross_validate(
+        df: pd.DataFrame,
+        task_type: str,
+        task_name: str,
+        probe_type: str,
+        probe_params: dict,
+        n_splits: int,
+        embedding_dim: int,
+        output_dir: Path,
+        filename_prefix: str,
+        enable_plots: bool,
+        random_seed: int = 42,
+        output_fold_results: Optional[bool] = False,
+        store_probes: Optional[bool] = False,
+        ) ->  TaskResult:
+    """Perform shuffle split validation with a linear probe. 
+    Returns the trained model from the last split in the return object.
+
+    Args:
+        df (pandas.DataFrame): Dataframe to evaluate.
+        task_type (str): Type of task, either "classification" or "regression".
+        task_name (str): Name of task.
+        probe_type (str): String with the probe type. Currently implemented are 'linear' and 'svm', the former using a single linear layer and the second an SVM.
+        probe_params (dict): Dictionary with parameters for probe.
+        device (torch.device): Device, CPU or GPU, to run training and inference.
+        n_splits (int): Number of repetitions of Linear Probe evaluation. Use to gather statistics.
+        embedding_dim (int): Size of embeddings.
+        output_dir (str): Path to folder to store results in.
+        enable_plots (bool): Toggle storing of plots. Set to True to store plots.
+        random_seed (int): Integer seed for random number generator.
+        output_fold_results (bool, optional): Toggle storing performance metric per fold in addition to summary statistics. Default is False, in which case performance per fold is not stored.
+        store_probes (bool, optional): Toggle storing model under <output_dir>/models/<task_name>. Default is False. The model from the final fold is stored.
+    Returns:
+        TaskResult: A TaskResult instance containing the evaluation results.
+    """
+
+    logger.info("Cross-validation start: %s", task_name)
+
+    assert probe_type in ['linear', 'svm'], f'Probe type must be "svm" or "linear", but was {probe_type}.'
+    assert output_dir is not None, f'output_dir cannot be empty.'
+
+    if probe_params is None:
+        probe_params = {}
+
+    splitter = ShuffleSplit(
+        n_splits=n_splits, test_size=0.1, random_state=random_seed
+    )
+    loss_curve_x_label = None
+    if probe_type == 'linear':
+        loss_curve_x_label = "Epoch"
+
+        # Determine device
+        device = probe_params.pop("device", "auto")
+        device = (
+            torch.device("cuda" if torch.cuda.is_available() else "cpu")
+            if device == "auto"
+            else torch.device(device)
+        )
+        
+        trainer = LinearFoldRunner(df=df,
+                                   task_type=task_type,
+                                   task_name=task_name,
+                                   device=device,
+                                   output_dir=output_dir,
+                                   filename_prefix=filename_prefix,
+                                   enable_plots=enable_plots,
+                                   embedding_dim=embedding_dim,
+                                   probe_params=probe_params,
+                                   splitter=splitter,
+                                   store_probes=store_probes,
+                                   )
+        
+        fold_results = trainer.train()
+        model = trainer.model
+        best_params = trainer.hyperparams
+
+    elif probe_type == 'svm':
+        loss_curve_x_label = "Hyperparameter setting"
+        
+        # Set random seed for optimizer if not already defined
+        opt_params = probe_params.get('opt_params', None)
+        if opt_params is not None:
+            if 'random_state' not in opt_params:
+                opt_params['random_state'] = random_seed
+                probe_params['opt_params'] = opt_params
+
+        trainer = SVMHPOptimizer(
+            df=df,
+            task_type=task_type,
+            task_name=task_name,
+            output_dir=output_dir,
+            filename_prefix=filename_prefix,
+            enable_plots=enable_plots,
+            splitter=splitter,
+            **probe_params,
+        )
+        fold_results = trainer.train()
+        model = trainer.model
+        best_params = trainer.best_model_state
+
+    # Aggregate and save results
+    train_losses = [fr.train_loss for fr in fold_results]
+    val_losses = [fr.val_loss for fr in fold_results]
+    best_metrics = np.array([fr.best_metric for fr in fold_results], dtype=np.float64)
+
+    save_loss_curve(train_losses, output_dir, task_name, loss_type="train", xlabel=loss_curve_x_label,)
+    save_loss_curve(val_losses, output_dir, task_name, loss_type="validation", xlabel=loss_curve_x_label,)
+
+    # Compute summary Q-statistic
+    mean_score = np.nanmean(best_metrics)
+    std_dev = np.nanstd(best_metrics)
+    q_stat = mean_score / (0.02 + std_dev) * 2
+
+    result_metrics = {
+        "q_stat": q_stat,
+        "mean_score": mean_score,
+        "std_dev": std_dev,
+        "hyperparamaters": best_params,
+    }
+    if output_fold_results:
+        result_metrics["q_t"] = best_metrics.tolist()
+
+    (output_dir / task_name / f"{task_name}_result.json").write_text(
+        json.dumps(result_metrics, indent=2)
+    )
+
+    # Store models
+    if store_probes:
+        model_save_path = output_dir / task_name / 'probe'
+        os.makedirs(model_save_path, exist_ok=True)
+        trainer.save_model(model_save_path)
+
+    return TaskResult(task_name, q_stat, mean_score, std_dev, model)