gym list (and gym search, which reuses it) takes ~10s because it runs the expensive Hydra/OmegaConf resolver over every benchmark config twice:
discover_benchmarks() calls BenchmarkConfig.from_config_path which builds each BenchmarkConfiglist_benchmarks() calls read_config_metadata() per benchmark to fetch domain and description information
Both call _parse_no_environment_tolerating_unset_values under the hood. Pass 2 is redundant: pass 1 already produced a resolved config that contains domain/description.
Proposed fix (low risk, ~halves runtime):
Capture domain and description during discovery -- call _scan_servers_for_metadata() on the already-resolved dict inside BenchmarkConfig.from_initial_config_dict, store both on BenchmarkConfig, and have list_benchmarks read them directly instead of calling read_config_metadata.
gym list(andgym search, which reuses it) takes ~10s because it runs the expensive Hydra/OmegaConf resolver over every benchmark config twice:discover_benchmarks()callsBenchmarkConfig.from_config_pathwhich builds eachBenchmarkConfiglist_benchmarks()callsread_config_metadata()per benchmark to fetch domain and description informationBoth call
_parse_no_environment_tolerating_unset_valuesunder the hood. Pass 2 is redundant: pass 1 already produced a resolved config that contains domain/description.Proposed fix (low risk, ~halves runtime):
Capture domain and description during discovery -- call
_scan_servers_for_metadata()on the already-resolved dict insideBenchmarkConfig.from_initial_config_dict, store both onBenchmarkConfig, and havelist_benchmarksread them directly instead of callingread_config_metadata.