Remove n_points > 1 option from best-point selector #3913
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context:
TLDR: I am cleaning up and refactoring best-point functionality.
n_points > 1
is not supported, so it doesn't make sense to refactor it.GeneratorRun
, then parsed byBestPointMixin._get_best_trial
, and then some more processing is done to ensure that the point is not None. This is OK for comparing different generation strategies, because it ensures that a best point will always be present. However, it not very helpful for comparing best-point selection itself. The right way to change a best-point recommendation is viaGeneratorRun.best_arm_predictions
, but all the post-processing will make it hard to see the effect of that. We initially wroteBenchmarkMethod.get_best_parameters
with the expectation that users could subclassBenchmarkMethod
and overrideget_best_parameters
in order to control best-point selection more directly. But subclassingBenchmarkMethod
is a bad idea that would cause serialization issues, and developing a best-point selector in that way would then require the work be redone to integrate it into Ax.GeneratorRun.best_arm_predictions
. If this is None, the inference trace will be NaN at that point.Differential Revision: D76159679