feat: Introduce NePS-Spaces #248

Meganton · 2025-10-23T12:52:15Z

This pull request updates the documentation to standardize and modernize how search spaces are defined and referenced in NePS. The main change is moving from dictionary-based search space definitions to using classes that inherit from neps.PipelineSpace.

Migration to class-based search space definitions:

The new spaces include HPO but also general CFG-like sampling capabilities. See the new docs for more information.
Updated all examples in docs/getting_started.md, docs/index.md, and docs/reference/neps_run.md to use a class inheriting from neps.PipelineSpace instead of a dictionary for defining the search space. This includes updating parameter definitions to use neps.Float, neps.Integer, and neps.Categorical with new argument names like min_value and max_value. [1] [2] [3]

Reference and terminology updates:

Updated method calls to use the new class-based search space, e.g., neps.run(evaluate_pipeline, ExampleSpace()) instead of passing a dictionary. [1] [2] [3]

Minor formatting and clarity improvements:

Clarified argument names and usage in code samples to align with the updated API. [1] [2] [3]

Merge

- Introduced tests for various aspects of the NEPS space, including: - `test_search_space__nos_like.py`: Tests for the resolution of a complex pipeline with unary, binary, and ternary operations. - `test_search_space__recursion.py`: Validates recursive operations within the pipeline, ensuring correct factor propagation. - `test_search_space__resampled.py`: Tests for resampling behavior in both float and integer pipelines, ensuring shared and unique values are handled correctly. - `test_search_space__reuse_arch_elements.py`: Verifies the reuse of architectural elements in pipelines, ensuring correct operation and configuration string generation. - `utils.py`: Added utility functions to generate possible configuration strings for pipelines, aiding in testing and validation.

…odule - Changed all instances of `space` to `neps_space` in test files to reflect the new import structure. - Updated class definitions to inherit from `neps_space.Pipeline` instead of `space.Pipeline`. - Modified the creation of operations and categorical choices to use `neps_space.Operation` and `neps_space.Categorical`. - Adjusted the resolution and conversion functions to utilize the new `neps_space` methods. - Ensured consistency across all test cases for the new structure and functionality.

- Added a new module `bracket_optimizer.py` that implements a bracket-based optimization strategy for multi-fidelity configurations in NePS spaces. - Introduced the `_BracketOptimizer` class to handle the optimization process, including sampling configurations from a prior band and managing different fidelity levels. - Implemented various bracket types such as successive halving, hyperband, ASHA, and async hyperband. - Updated `priorband.py` to utilize the new sampling strategies and refactored the prior sampling logic. - Created a new `sampling.py` module to define various samplers, including `OnlyPredefinedValuesSampler`, `RandomSampler`, `PriorOrFallbackSampler`, and mutation-based samplers. - Modified existing tests to accommodate the new optimizer and sampling strategies, ensuring compatibility with the updated structure. - Refactored import paths in test files to align with the new module organization.

- Updated import statements to use `neps.space.neps_spaces.parameters` for all relevant classes and functions in test files. - Refactored test classes and methods to replace direct usage of `neps_space` with the new parameters module, ensuring consistency across the test suite. - Adjusted the instantiation of various operations and categorical choices to align with the new structure. - Ensured that all references to operations, floats, integers, and categorical choices are correctly sourced from the updated parameters module.

- Updated imports in test files to directly import necessary classes from neps.space.neps_spaces.parameters. - Refactored class definitions to inherit directly from Pipeline and other relevant classes. - Replaced instances of parameter class references with direct class references for improved readability. - Ensured consistency across test files by applying similar refactoring patterns.

…pipeline adjustments

- Updated imports for optimizers in test_neps_integration.py to use the new neps.optimizers module. - Replaced references to old RandomSearch and ComplexRandomSearch algorithms with their new counterparts from neps_algorithms. - Adjusted the optimizer imports in test_neps_integration_priorband__max_cost.py and test_neps_integration_priorband__max_evals.py to align with the new structure. - Ensured all instances of priorband optimizers are updated to use the new neps_algorithms module.

…ules

…dules

…eration classes to improve clarity and consistency

…arity and consistency - Updated docstrings in SamplingResolutionContext to use Args and Returns sections for better readability. - Changed parameter names from `prior_index` to `prior` in Categorical class to enhance clarity. - Introduced a new function `convert_confidence_level` to handle string to ConfidenceLevel conversion. - Updated tests to reflect changes in parameter names from `prior_index` to `prior`. - Enhanced docstrings across various classes and methods in the sampling module for consistency.

… and convert_confidence_level function

…es and enhancing links to related topics

…ing clarity, and improving formatting in code examples

… HPO classes - Updated the image segmentation example to utilize HPO_Float and HPO_Integer for hyperparameter definitions. - Refactored test cases to replace old parameter classes with HPO variants, ensuring compatibility with the new hyperparameter optimization framework. - Added log sampling for NePS-Integer - Adjusted imports in test files to reflect changes in the neps library structure. - Enhanced the configuration encoder tests to validate the new HPO parameter types. - Ensured all relevant tests are passing with the updated hyperparameter definitions.

…md file - Corrected links in getting_started.md, evaluate_pipeline.md, and neps_run.md to point to neps_spaces.md. - Updated navigation in mkdocs.yml to reflect the removal of pipeline_space.md. - Enhanced clarity in algorithms.py and grid.py by updating parameter references. - Added validation in parameters.py to ensure prior_confidence is set when prior is defined.

…n neps.run() and enhance fidelity parameter explanation

…hods

…with pre-commit changes

…into neps-spaces-updates

…prior method in PipelineSpace

…ter readability and maintainability

…cking and improve readability

…thods to avoid using a cached resolved object instead of resolving when they are actually different

…ing criteria

…y when required and improve DataFrame handling

…ts for consistent trace text generation

…s for better organization

…aching

… metrics functionality - Introduced tests for converting between classic SearchSpace and NePS PipelineSpace, ensuring attributes and properties are preserved. - Implemented tests for algorithm compatibility with both NePS-only and classic algorithms. - Added tests for trajectory and metrics functionality, including handling of fidelity parameters and cumulative metrics tracking. - Included error handling tests for failed evaluations and validation of metric values. - Verified file format and structure for trajectory and best config outputs. - Ensured that NePS can revisit previous runs and update trajectories correctly.

…cripts - Deleted `priors_test.ipynb` and `warmstarting.py` as they were no longer needed.

…egration tests for hyperparameter optimization with fidelity

-> Add trial validation and import functionality - Introduced `TrialValidationError` exception for handling trial configuration validation errors. - Created a new module `normalization.py` for normalizing imported configurations. - Implemented `import_trials` method in various optimizers (e.g., `BayesianOptimization`, `BracketOptimizer`, `GridSearch`, `IFBO`, `PriMO`, `RandomSearch`) to handle external evaluations. - Added validation functions in `validation.py` to ensure imported results and configurations meet expected criteria. - Enhanced `Trial` class to support loading from external data with a new `load_from_dict` method. - Updated `BudgetInfo` and `WorkerSettings` to allow for float values in `fidelities_to_spend`. - Added utility functions in `trial_io.py` for loading trials from pickle files. - Created an example script `example_import_trials.py` to demonstrate the import trials functionality.

… for improved sampling accuracy

…lity

nastaran78 · 2025-10-27T11:10:31Z

neps/optimizers/utils/grid.py

+    HPOInteger,
+    SearchSpace,
+)
+from neps.space.neps_spaces import neps_space


Nit: if we are exposing the specific functions from neps_space to user-side, it's it better to make it accessible from neps directly? if it's a util function which is not the main functionality and not used in the happy path but will give insight to user I'd suggest to put it in neps.utils

Which specific functions do you mean should be moved to util?

but moved, but maybe imported in the init of utils package, I was thinking that convert_operation_to_string is not something that user may need normally.

nastaran78 · 2025-10-27T11:21:03Z

neps/space/neps_spaces/parameters.py

+          ValueError: If the resampling is by name or the source is not resolvable.
+
+        """
+        if self.is_resampling_by_name:


DRY: the validation part for both get_attrs and from_attrs are the same, potentially we could extract a _get_resolvable_source function.

nastaran78 · 2025-10-27T11:22:47Z

neps/space/neps_spaces/parameters.py

+        return f"Resampled({self._source!s})"
+
+    @property
+    def source(self) -> Resolvable | str:


Nit: we can get rid of the source property and rename _source to source if we don't have extra steps here in getter.

nastaran78 · 2025-10-27T11:56:38Z

neps_examples/basic_usage/pytorch_nn_example.py

+            (Resampled("model"), Resampled(_C)),
+            (Resampled(_O), Resampled(_O), Resampled(_O)),
+            (
+                Resampled("model"),


I suppose this has a high potential for infinite recursion. Python's call stack exceeds its limit (1000 by default). I encountered that 2 times (however the odds should be low by theory). Maybe we can validate that our grammar is a DAG?

No, we specifically want these recursions to be possible. What does Neps do when encountering an error during sampling? Would it just resample?

we have a 10 time retry in the runtime function when catching an exception.
I don't get any Neps error, I get the error from system, which we cannot prevent if we can't have a validation like that, that's sad.

nastaran78 · 2025-10-27T12:43:30Z

neps_examples/basic_usage/pytorch_nn_example.py

+import torch
+import torch.nn as nn
+import neps
+from neps.space.neps_spaces.parameters import (


use the exposed neps.PipelineSpace

nastaran78 · 2025-10-27T13:12:48Z

neps/optimizers/algorithms.py

+        converted_space = convert_neps_to_classic_search_space(pipeline_space)
+        if converted_space is not None:
+            pipeline_space = converted_space


Suggested change

converted_space = convert_neps_to_classic_search_space(pipeline_space)

if converted_space is not None:

pipeline_space = converted_space

pipeline_space = convert_neps_to_classic_search_space(pipeline_space)

if pipeline_space is None:

raise ...

I'm not sure if this works everywhere, as we sometimes proceed with the old space to run an old optimizer. I think it's better to keep them separated.

I understand your point, but here we are assigning that to pipeline_space anyways if it's not None and otherwise raising error.

nastaran78 · 2025-10-27T13:36:24Z

neps/space/neps_spaces/neps_space.py

+                            value.prior_confidence.value if value.has_prior else "low"
+                        ),
+                    )
+                elif isinstance(value, Fidelity):


I think it doesn't reach this condition, because we don't have the Fidelity attributes in get_attrs().
I believe, we should separately iterate over fidelity_attrs().

True, i missed that, thanks. Don't understand why it worked anyway, but yeah we can just call fidelity_attrs and add it to the dict we're iterating over

nastaran78 · 2025-10-27T16:09:23Z

neps/space/neps_spaces/neps_space.py

+            environment_values=sampled_pipeline_data.environment_values,
+        )
+
+        config = dict(**sampled_pipeline.get_attrs())


I think it's better to have get_attr more general containing fidelities. maybe we can have a flag as an argument whether we want the fidelity parameters or not.
here also I think our intention is to include fidelity.

lumib and others added 30 commits April 14, 2025 18:44

Merge pull request #1 from automl/master

dec8a1b

Merge

Initial code for the new pipeline space

4ea7acb

Add centering

5bbbce8

Add bracket optimizers

a6b10ff

WIP

6703974

Allow having Operation args and kwargs themselves be Resolvable objects

9cbcc97

Add NOS like space

ed417fe

Minor change in config pretty print

0ad5ac1

Merge remote-tracking branch 'upstream/master'

6ce6507

Merge branch 'master' into new-pipeline-space

e72a3d0

Refactor NEPS space integration tests to remove redundant evaluation …

3b58495

…pipeline adjustments

Add Pytorch Neural Network example

9bc6ca8

Change import order

eaceb8c

Refactor type hints and improve docstring formatting in optimizer mod…

917939b

…ules

Refactor documentation and remove redundant arguments in optimizer mo…

bc89f65

…dules

Enhance docstrings for Resolvable, Fidelity, Pipeline, Domain, and Op…

5b68925

…eration classes to improve clarity and consistency

Add documentation for NePS Spaces framework and usage examples

8c5e9f0

Fix indentation in docstring for clarity in SamplingResolutionContext…

a8f641a

… and convert_confidence_level function

Improve documentation clarity in NePS Spaces by refining section titl…

2243da7

…es and enhancing links to related topics

Refine documentation in NePS Spaces by correcting terminology, enhanc…

b6a7b30

…ing clarity, and improving formatting in code examples

Add neps_algorithms as direct sub-import of neps

0a5a7e1

Meganton added 21 commits September 21, 2025 19:08

Update NePS Spaces documentation to clarify usage of pipeline_space i…

0d9fafc

…n neps.run() and enhance fidelity parameter explanation

Merge branch 'master' into neps-spaces

464bae7

feat: Enhance NePS Spaces with new parameter addition and removal met…

adfefe1

…hods

feat: Add methods for adding and removing parameters in NePS Spaces

0ab0e09

Started HyperBand

6e3faa9

Introduce NePS-space compatible Grid Search and Hyperband variants

cfb4b11

Introduce NePS-space compatible Grid Search and Hyperband variants - …

db37873

…with pre-commit changes

Merge branch 'neps-spaces-updates' of https://github.com/automl/neps …

076b07d

…into neps-spaces-updates

feat: Enhance DefaultWorker to return cumulative metrics and add add_…

6f7011d

…prior method in PipelineSpace

feat: Refactor trace text generation into a reusable function for bet…

97be100

…ter readability and maintainability

feat: Refactor load_incumbent_trace to include cumulative metrics tra…

a2c82a7

…cking and improve readability

feat: Enhance SamplingResolutionContext with path-aware resolution me…

cb02744

…thods to avoid using a cached resolved object instead of resolving when they are actually different

feat: Update best_config.txt with final cumulative metrics upon stopp…

eb985b0

…ing criteria

feat: Update summary writing logic to ensure metrics are recorded onl…

85916f4

…y when required and improve DataFrame handling

feat: Refactor trajectory_of_improvements to utilize _build_trace_tex…

86a3775

…ts for consistent trace text generation

feat: Move _build_trace_texts function to status.py and update import…

69d6eb5

…s for better organization

feat: Remove trajectory_of_improvements function

34af3aa

feat: Fix resampling by seperating __eq and is_equivalent. Improved c…

740065a

…aching

Remove old warmstarting functionality, unused example notebooks and s…

4ed7bda

…cripts - Deleted `priors_test.ipynb` and `warmstarting.py` as they were no longer needed.

feat: Refactor pipeline space definition into a class and add new int…

89f3346

…egration tests for hyperparameter optimization with fidelity

github-project-automation bot added this to NePS Project Board Oct 23, 2025

Meganton assigned Meganton and nastaran78 Oct 23, 2025

Meganton added 2 commits October 26, 2025 23:28

fix: update confidence level in MutatateUsingCentersSampler to MEDIUM…

0c519ce

… for improved sampling accuracy

fix: improve formatting in wrap_config_into_string for better readabi…

d83040d

…lity

nastaran78 reviewed Oct 27, 2025

View reviewed changes

Uh oh!

feat: Introduce NePS-Spaces #248

Are you sure you want to change the base?

feat: Introduce NePS-Spaces #248

Uh oh!

Conversation

Meganton commented Oct 23, 2025

Uh oh!

nastaran78 Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastaran78 Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nastaran78 Oct 27, 2025 •

edited

Loading

nastaran78 Oct 27, 2025 •

edited

Loading