Refactor hf-sim to use xarray and Dask for larger-than-memory station processing

[Copilot]

Summary

Refactors the hf-sim script to replace ThreadPoolExecutor with xarray.map_blocks + dask.distributed for parallelization, enabling scaling to 100k+ stations and supporting larger-than-memory output files (up to 100GB+). The entire Dask graph flows lazily to to_netcdf, so the full waveform array never needs to reside in memory.

Changes

New Functions

• load_hf_dataset(...) — Reads the station CSV, computes seeds and vref, returns a chunked xr.Dataset indexed by station. Chunk size is calculated as max(1, total_stations // 500) to keep the task graph between ~500–1000 tasks.
• process_hf_dataset(ds, *, hf_sim_path, hf_input_template) — map_blocks-compatible function that iterates stations in a chunk and calls hf_simulate_station. Returns waveform (component × station × time) and epicentre_distance (station).

CLI Refactor

• Workflow: load_hf_dataset → build template with dask.array.empty → xr.map_blocks(process_hf_dataset) → assign metadata lazily → ds.to_netcdf(out_file, engine="h5netcdf").
• No eager .compute(), .result(), or .values on large data — Dask streams chunks to disk one at a time.
• Uses LocalCluster for parallel subprocess execution within the container.

Dependencies

• Added dask[distributed] to pyproject.toml.

Tests

• test_load_hf_dataset — Checks dataset structure, coordinates, and attributes.
• test_load_hf_dataset_chunking — Verifies chunking logic with 1500 stations (chunk_size=3, no data loss).
• test_process_hf_dataset_structure — Mocks hf_simulate_station and verifies output dataset dimensions.

Notes

• hf_simulate_station and process_hf_dataset are standalone (not nested) functions for pickle serialization to Dask workers.
• Dask worker memory limits should account for the external hb_high_binmod binary's memory footprint (documented in module docstring).
• The map_blocks template uses dask.array.empty to avoid allocating memory for the full waveform array upfront.

[lispandfound]

I've changed my mind here. This is run in a container, so a slurmcluster is a bad idea because it requires too many hacks. However I do believe this template is still useful because dask allows us to process larger than memory datasets which multiprocessing won't do. Take out the slurm stuff and refactor this to focus on being able to process larger-than-memory outputs. This means in particular you should not eagerly compute the whole array at any point (no .values, no .result, no .compute) and instead allow the graph to get all the way up to to_netcdf before saving to let dask handle the memory pressure. These output files can be up to 100gb.

[Copilot]

I've changed my mind here. This is run in a container, so a slurmcluster is a bad idea because it requires too many hacks. However I do believe this template is still useful because dask allows us to ...

Done in 9953992. Removed all SLURM/HostType/dask-jobqueue code and the --host CLI option. The pipeline is now fully lazy:

• Template uses dask.array.empty so no memory is allocated upfront for large waveform arrays
• No .compute(), .result(), or .values on large data — the Dask graph flows from map_blocks all the way to to_netcdf
• to_netcdf streams chunks to disk one at a time, so 100GB+ output files never need to reside fully in memory
• LocalCluster is used for parallel subprocess execution within the container

[lispandfound]

You should use xarray testing facilities to simplify this test

[Copilot]

Refactored in 9b69839. test_load_hf_dataset now builds an expected xr.Dataset and uses xr.testing.assert_allclose for the data variable comparison. test_process_hf_dataset_structure uses xr.testing.assert_equal for coordinate comparisons instead of np.testing.assert_array_equal.