Skip to content

CLAUDE.md

Latest commit

 

History

History
531 lines (486 loc) · 40 KB

CLAUDE.md

File metadata and controls

531 lines (486 loc) · 40 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

infrastore is a Rust library for managing time-series data in power-systems / energy simulations. Persistence is split between numerical arrays in HDF5 and metadata associations in SQLite. It exposes multiple bindings over a shared core:

  • Native Rust — infrastore-core public API
  • gRPC server + Rust client — infrastore-server (read-only server; writes need local filesystem access)
  • Python — infrastore-py via PyO3 (abi3-py311 wheel)
  • Julia — infrastore-ffi C ABI cdylib, wrapped by julia/InfraStore.jl
  • CLI — infrastore-cli (infrastore binary): loads time series from CSV + a descriptor JSON and inspects a store, talking directly to the on-disk HDF5 + SQLite artifact (read+write; no gRPC). Output uses a global -f/--format table|json|jsonl|csv|parquet.
  • Parquet — infrastore-parquet, a crate the CLI depends on behind a parquet cargo feature that is on by default, so the shipped infrastore binary carries it; the line Arrow must not cross is into the libraries — infrastore-core, infrastore-py, and infrastore-ffi never link it, and cargo tree --edges normal on each is the check. The feature stays switchable (--no-default-features --features vendored), and a binary without it still parses -f parquet and --parquet and names the feature to rebuild with. export -f parquet --dir writes a normalized, partitioned layout — per (time_series_type, value type, time_reference) triple, two files sharing a stem: <stem>.values.parquet holds every distinct array once, one row per value, and <stem>.series.parquet holds one catalog row per series. Both carry the array key (data_hash, time_axis) and are sorted by it. The triple partitions because those three cannot vary inside one table without nullable or ill-typed columns; the payoff is that every column is required, which the five free-form descriptors pay for with the empty string. The split is because the store is content-addressed: a thousand components sharing one profile hold one array, and a denormalized table would write it a thousand times. time_axis spells whatever decides a series' timestamps for its type (a repeating interval for a grid, the timestamps_hash for an irregular axis, count/interval/horizon/resolution for a forecast) and is read off the values exported, not the catalog row. Composite kinds partition by kind alone and are re-padded to the partition's widest series, so their data_hash is taken over the decoded points — which is also what lets two paddings of one curve share a values group. add --parquet takes a file, a directory, or a partition stem and reads the pair as a merge join, one transaction per partition, with a dangling key on either side an error and --no-checksum waiving the data_hash check. A values file with no series file beside it is a foreign file. An empty series fails the export, naming every one and writing nothing. Python's to_arrow()/from_arrow() are per-series in-memory conveniences, not this format — "one schema, two producers" was withdrawn. See docs/src/reference/parquet-format.md.

Current feature coverage: SingleTimeSeries, NonSequentialTimeSeries, and PersistentTimeSeries are implemented end-to-end (read+write in the Rust core, C ABI, Python, Julia, and the CLI; read-only over gRPC). A PersistentTimeSeries is a sparse step function: breakpoints plus one value each, where the value at an instant is the one belonging to the greatest breakpoint <= t — held forward past the last breakpoint and undefined before the first, which is an error rather than a clamp. It is structurally identical to NonSequentialTimeSeries and shares its storage (PackGroup is keyed by the time axis, never by the type, so the two pool into one nsts_… dataset and dedup arrays against each other); the difference is entirely in read semantics, which is why it is a distinct type rather than a read flag — "an irregular timeline has no value between its timestamps" is a guarantee NonSequentialTimeSeries leans on. A range read begins at the breakpoint in force at start, so the result always defines a value there. Scalar-collapse policy is the application's and rides in application_data; there are no catalog columns for it. It is an infrastore-local extension, not a Sienna type: it sits outside the vendored six-type oneOf, nothing under conformance/ mentions it, and it does not travel in an OpenAPI document in either direction — the export omits its rows (refusing a filter that names the type outright) and the import rejects one a foreign document carries. Deterministic, DeterministicSingleTimeSeries, Probabilistic, and Scenarios support reading values across the Rust core, C ABI, Python, Julia, and gRPC. Dense forecasts (Deterministic, Probabilistic, Scenarios) are written through the generic add_time_series by passing the matching forecast object across the Rust core, Python, and Julia (the C ABI keeps per-type infrastore_store_add_forecast / infrastore_store_add_probabilistic as low-level transport); DeterministicSingleTimeSeries is derived from stored SingleTimeSeries via transform_single_time_series rather than added directly. Forecast writes are not exposed over the read-only gRPC server. Arrays are dtype-generic (f64/f32/i64/i32/u64/bool in every binding, including Python) and may have multidimensional per-timestep values. The columnar simulation readers (StaticReader/ForecastReader) are bound across the Rust core, C ABI, Julia, and Python; StaticReader covers all three static types, sweeping a SingleTimeSeries grid, a cohort of NonSequentialTimeSeries sharing one timestamp vector, or a set of PersistentTimeSeries (its resolution() is None for both irregular kinds). The persistent case is the one exception to "one timeline per reader": a step function has a value at every instant from its own first breakpoint on, so its columns may hold independent breakpoint vectors. Such a reader interns the distinct vectors, gives each column the id of the one it resolves against, and takes their sorted union as its public axis; index_at then reports a position on that union and is not a storage row index. There is still no presence mask — an instant before some column's first breakpoint is a hard error naming that column. A SingleTimeSeries reader need not inherit its grid either: Store::build_static_reader_over(filter, ReadWindow) (window_start= / window_length= in Python and Julia, --window-start / --window-length on the CLI's grid, has_window_start and friends across the C ABI) takes a caller-named span, and each column then reads at a row offset of its own — so series with different starts or lengths sweep together. The axis is still single; only its provenance changes. All three edges are checked at build: a matched series that does not cover the span is an error naming it rather than a column silently dropped, the anchor is checked against each column's grid rather than floored onto it (unlike read_by_id, which floors because a value covers its step), and a calendar period meets the same Period::sub_grid_is_anchorable rule a slice does, since a window is a re-anchoring. A window whose columns all start at the anchor falls back to the single-index backend read, so the uniform sweep is unchanged. The other remedy for a divergent selection is ListFilter::initial_timestamp + ListFilter::length, which with resolution name a whole grid and match only the series already on it (initial_timestamp= / length= in Python and Julia, --initial-timestamp / --length as CLI selectors, a has_-flagged pair on every filter-taking C ABI export). It is the constructive half of the grid-coherence rule, the role ListFilter::zoneless plays for spelling. The two answer different questions and compose: a window sweeps a named span across the ragged series, the filter drops the ones that are not on the grid. Being a filter it selects rather than asserts — a grid no row is on is an empty result, not an error, and a row storing no initial_timestamp (the two irregular types) matches no value at all. Neither is part of KeyIdentity, so an identity probe never narrows by them: two series differing only in start or length are the same row to the catalog.

The discovery/maintenance surface (get_intervals, list_names, list_owner_types, name-pattern filtering via ListFilter::name_glob (SQLite GLOB), ListFilter::component_field (exact match; served by the partial index idx_component_field, so it can never select rows that left the field unset), remove_by_filter, the time-sliced read_by_ids_range, AddRequest/Store::add preserving application_data, and serde on the core types) is available in the Rust core and threaded through the C ABI/Julia and Python bindings. Two association catalogs are available in the Rust core, C ABI, Julia, Python, and the CLI (read via attributes / links, write via attach / detach / link / unlink / reassign), but not over gRPC: supplemental_attribute_associations (component ↔ supplemental attribute, the wider surface — counts, counts-by-type, grouped summary) and parent_child_associations (directed component ↔ component edges, e.g. a generator connected to a bus, deliberately narrower until a consumer needs more). Both are independent of time series in both directions, and of each other. Beside them sits store_attributes, free-form key/value provenance about the artifact rather than a row (who built it, from what source system, under which of the consumer's own schema versions) — never interpreted, TEXT values, one value per key so a set is an upsert, infrastore. reserved on removal as well as on write, and probed by is_empty because those rows are the consumer's own text and recoverable nowhere else. Available in the Rust core, C ABI, Julia, Python, and the CLI (store-attr, plus store_attributes in store-info), with ListStoreAttributes / GetStoreAttribute as the gRPC read half. The store_ prefix is load-bearing: a bare "attribute" means a supplemental attribute here and a bare "metadata" means a TimeSeriesMetadata row. Every catalog row carries an id — an INTEGER PRIMARY KEY AUTOINCREMENT, so it is never reissued once its row is deleted — and it is the only way to address a stored time series. A consumer records the id in its own object model and references the series by it (a generator's operation_cost naming the series that varies it). In the Rust core it is the newtype TimeSeriesId(i64), so an owner_id cannot be passed where a series id belongs; it is #[serde(transparent)], so SQLite, the gRPC wire and the OpenAPI document are unchanged and every binding still exchanges a plain integer.

The surface splits into identify and act. Identifying is four calls, all returning the same TimeSeriesMetadata row: list_metadata(filter) (by attributes, 0..N), list_metadata_by_ids(ids) (ordered, NotFound on any stale reference), get_metadata_by_id(id) (None for a stale one, because a consumer validating stored references is asking a question), and association_exists(id) (a primary-key probe that fetches no row). ListFilter is the single identity vocabulary and the flexible half of the split — there is deliberately no separate attribute-to-id resolver. Acting takes ids: read_by_id(id, ReadWindow) is the single-id read and the sliced one (a ReadWindow of start/len/count resolved against the row the primary-key lookup already returned, checked rather than clamped, so a keyed read costs one call); read_by_ids(ids, ReadWindow) is its bulk form and read_by_ids_range(ids, TimeRange) the bounds form that clips instead of checking (what an export wants, since it knows the bounds and not the step count) — with three type-specific rules: a SingleTimeSeries start inside a step is floored onto the grid, because a value covers its step, while a NonSequentialTimeSeries selects only timestamps at or after start; a PersistentTimeSeries begins at the breakpoint in force at start, so the result always defines a value there; and a forecast's start must be a window boundary at or before the last window, since there is no partial window to return, so only its end clips — except a start before the first window, which clips too, because nothing partial lies there and refusing it failed every export window wider than the data. Cutting across all of them: a calendar period is not closed under slicing. A series is stored as an anchor plus a period and a count, and Period::Months clamps to month end non-associatively, so a monthly grid from Jan-31 (Jan-31, Feb-29, Mar-31) re-anchored at its own Feb-29 reads Feb-29, Mar-29, Apr-29 — right values, wrong dates, no signal. No anchor fixes it, so such a slice is refused (InvalidParameter) by both read forms, and transform_single_time_series is held to the same rule at write time since each derived window is a run of the source's own steps described that same way. Period::sub_grid_is_anchorable is the predicate. remove_by_ids / remove_by_ids! is all-or-nothing; copy_time_series(id, …) takes one. A series' name is fixed once written — there is no rename, so an id and the row it names can never drift apart. Writes return ids and nothing else — TimeSeriesId / Vec<TimeSeriesId> in the Rust core, int in Python and Julia, an out_id across the C ABI — and a caller wanting the rest of the row asks get_metadata_by_id. Removals are not on the read-only gRPC server.

The id crosses the gRPC wire and the OpenAPI one — where the schema spells it association_id, a rename openapi.rs applies the same way it maps unit_system between the store's snake_case and the schema's SCREAMING_CASE. It is descriptive — outside a series' KeyIdentity and both content hashes — but unlike the descriptors above it describes the row rather than the data: it is per-store, so merge assigns fresh ids and diff ignores it, while reassign/compact/persist_to all preserve it. KeyIdentity — the uniqueness tuple the catalog files a row under — survives as an internal write-path type and is explicitly not an address. No add_* accepts an id — not add_time_series, a bulk add, or either association catalog's attach/link — because "never reissued" is a guarantee of AUTOINCREMENT, and a caller free to name an id could re-file a retired one. The single exception is the rows-only import import_time_series_associations_openapi (Store::import_association_rows), which files each row under the association_id the document recorded so its references survive: all-or-none across the batch, and only above the catalog's high-water mark. It refuses a row whose array is absent, a DeterministicSingleTimeSeries whose source SingleTimeSeries is neither in the document nor already stored, and an irregular row that does not name its time axis. That last one is why NonSequentialTimeSeries.timestamps_uri exists: the values cannot imply the axis, because arrays are content-addressed, so two irregular series with identical values on different axes share one stored array and only the catalog's timestamps_hash tells them apart. The wire form therefore locates the axis, and the import resolves it against the store (the axis ships in the array file, like the arrays). A PersistentTimeSeries row never gets that far: the type is an infrastore-local extension, outside the six the vendored contract defines, so the schema check refuses a document naming one — and no export writes one either, which leaves the rows-only Store::import_association_rows as the only door such a row fits through. Neither association catalog's wire form carries an id, so both always assign — their row types carry an id field that a listing populates and an add ignores — with independent counters, and equality on both association types deliberately excludes the id.

Both imports validate every incoming row against the vendored SiennaSchemas specs before decoding it (crates/infrastore-core/src/openapi/schema.rs, schemas at crates/infrastore-core/sienna_schemas/, embedded with include_str! so there is no filesystem or network access — refresh with scripts/sync_sienna_schemas.sh). A time-series row is checked against the per-type schema its own time_series_type selects, not the oneOf wrapper, so an error names the offending field instead of only reporting that nothing matched.

Together with Store::open_without_catalog — which opens the array half of an artifact whose .sqlite is absent, minting an empty catalog stamped to match the arrays — these make an artifact readable back from arrays plus a document alone, which is what a consumer shipping its own JSON (PowerSystems' system.json + time_series.h5) wants — all six time-series types included. It refuses (StoreExists) a catalog that is already there. Exposed across the C ABI, Julia (open_store_without_catalog), and Python (Store.open_without_catalog); see crates/infrastore-core/tests/json_only_restore.rs.

Metadata getters surface element_shape and features in every binding. Alongside units, a series carries two further unit descriptors in every binding: quantity_kind (free-form, QUDT QuantityKind local names recommended — it separates active from reactive power, which dimensional analysis cannot, and is the only record of what per-unit values measure) and unit_system (natural_units | component_base, a label the store never acts on; unset means unspecified, not natural units). A series also carries component_field (free-form; names the field on the owning component whose value these values are the time-varying form of, e.g. max_active_power — it records what the values are for, where name only says which series they are; it is the one descriptor that is also a filter, in every binding). All three are descriptive, so they sit outside a series' KeyIdentity and outside both content hashes, alongside application_data — the opaque package-owned payload formerly spelled ext.

Every series also carries a time_reference (TimeReference: Utc | FixedOffset(minutes) | Zone(iana_name) | Zoneless; None means unspecified), recording how its timestamps were spelled so a read hands back what a write declared instead of relabeling everything UTC. Each binding infers it from the input type — Python from tzinfo (naive → Zoneless, a key- bearing ZoneInfo → Zone), Julia from DateTime vs ZonedDateTime (FixedTimeZone vs VariableTimeZone in InfraStoreTimeZonesExt), the CLI from the text plus --assume-timezone / --zoneless; a native Rust caller declares it. It is descriptive like the three above (outside the key and both hashes, so two series differing only in it are a duplicate), but it is not inert: query bounds must match the series' spelling (TimeRange carries a zoneless flag and the core refuses a mismatch rather than coercing), and a selection spanning both coherence groups is refused by read_by_ids / read_by_ids_range and build_static_reader, with ListFilter::zoneless (--spelling zoned|zoneless in the CLI) as the constructive remedy. A reference is a spelling, not a grid: Period::Months still steps on the UTC calendar (warned about when it meets a zoned reference), and a local-clock grid belongs in NonSequentialTimeSeries. The core validates a zone name's shape only and never resolves it — no tz database; existence is audited by the layers that have one (the CLI via chrono-tz, Python via zoneinfo, Julia via TimeZones) and reported by store-info. The CLI is the one place that runs local → instant, so --assume-timezone <IANA name> refuses the skipped and repeated wall clocks per row rather than guessing. Python ships type stubs (infrastore.pyi + a pytest drift guard), a full exception hierarchy, keyword-only optional arguments, and the paired element-value forms the Rust core has — a from_values classmethod on each of the six types that encodes the values and declares the element type they imply (inferring it from the shape of a row, since a Python payload carries no type tag; element_type= there is an assertion, not an override) and .decoded_values() on a series, which takes both the element type and the leading-axis count off the series. encode_element_values / decode_element_values remain the lower-level pair, and the only way to name the one series from_values cannot: an empty tuple(N,f64), whose arity lives in rows it does not have. Julia returns its catalog/metadata/summary query results as structs (TimeSeriesMetadata, StaticGrid, … — see docs/src/reference/julia-api.md#result-types), overloads Base (==/hash/show/length/iterate on the value types), and offers do-block Store/open_store forms. It also carries the element-value codec — encode_element_values/decode_element_values over LinearFunction, QuadraticFunction, PiecewiseLinear, PiecewiseStep — held to conformance/element_type_vectors.json like the Python and TypeScript ones. Its value types are permissive where a consumer's domain types are strict (a zero- or one-point curve is a row the store accepts, so the codec must represent it), and named for the wire vocabulary so they cannot clash with InfrastructureSystems.jl's; a consumer decodes straight into its own types through the types keyword and extends element_type_tag/element_row_width/write_element_row! to encode from them. The write and read paths use it, so a series of domain values round-trips as those values: a constructor names the element_type from what it is given (a contradicting element_type= is an error, not an override), encoding happens at the ABI boundary so the struct keeps the values, and a read decodes — raw=true hands back the packing instead. A composite row's time_series_type names the decoded values, so it is one rank lower than the stored array. The readers stay raw: StaticReader/ForecastReader are the per-timestamp path and StaticGroup.dtype is physical. A TimeSeriesMetadata's time_series_type is the full Julia type, parameterized {T,N} off the row's own element_type/element_shape, so it equals typeof(read_by_id(...)) for every stored type but the derived one — a DeterministicSingleTimeSeries row keeps its own tag while a read hands back the Deterministic it becomes, parameterized alike (ask which kind a row is with <:, not ==); the counts and summaries group by stored type alone and stay bare. Every type-taking call — a time_series_type= filter, has_time_series, both readers — accepts either spelling and ignores the parameters, since identity carries no element type, so a row round-trips back into them. A stored DeterministicSingleTimeSeries always reads back as a Deterministic (storage-level view, by design); the DST tag remains visible in catalog surfaces (metadata rows, counts). The CLI additionally has export (bulk read-direction inverse of add; its timestamped CSV is re-readable by add, which detects the layout from the header), arrays / store-info and the data_hash + resolved HDF5 dataset/column on list/info, --name-glob selectors, --dry-run on destructive commands, store-creation --compression flags, shell completions, and a INFRASTORE_STORE env fallback. It also carries a wide-CSV ingest ("layout": "wide" plus an owner_map/owner_id_from column→owner mapping) and its inverse grid, which drives the core's StaticReader; discovery commands (names, owner-types, owners, exists); charting (get --plot sparklines and plot --kind line|duration|heatmap|fan|overlay, rendered by the hand-written src/chart/ SVG backend — deliberately no charting dependency, because deny.toml makes one a policy decision); diff and merge between two stores; init and --catalog attached|in-memory; and an inline flag form of add alongside --descriptor - (stdin), --dry-run, --replace, and --batch-size. A --endpoint mode pointing the read commands at the gRPC server is still the one documented gap; src/store_access.rs is the seam reserved for it. The SQLite catalog carries a time_series_readable view that hex-encodes both hashes for hand inspection. The read-only gRPC server carries the full read surface too, id-addressed like the rest: ListMetadata / ListMetadataByIds (rows each carrying their id), GetMetadataById, AssociationExists, HasAnyTimeSeries, ReadById / ReadByIds, detailed/per-type counts, ListOwnerIds, GetIntervals, static/forecast summaries, and CheckStaticConsistency. Every RPC is named for the Store method it exposes, with <Rpc>Req / <Rpc>Resp messages. Auth is none (default) or api_key via the x-api-key header. See README.md, docs/src/explanation/time-series-types.md (the six types and their write paths), and docs/src/explanation/bindings.md (per-binding coverage) for the authoritative feature matrix.

Code Quality Requirements

All code changes must pass the following checks before being committed:

cargo fmt --all -- --check                              # Rust formatting
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features                   # Tests
dprint check                                             # Markdown formatting
cargo deny check --config deny.toml                     # Dependency policy

Key requirements:

  • Rust code: Must compile without clippy warnings. Use cargo clippy --workspace --all-targets --all-features -- -D warnings to verify.
  • Toolchain: The workspace uses Rust edition 2024 and declares an MSRV of Rust 1.94 (see rust-version in the root Cargo.toml, which is the sole authority — there is no rust-toolchain file). Do not use APIs requiring a newer compiler without intentionally updating rust-version and CI.
  • Pre-commit: cargo-husky installs .cargo-husky/hooks/pre-commit, which runs rustfmt, Clippy, dprint, and shellcheck when available. Do not bypass a failing hook. Tests and cargo-deny are still required before committing.
  • CI: Workspace builds and tests run on Linux, macOS, and Windows. Avoid Unix-only assumptions in shared Rust code, build scripts, paths, and workflow changes.
  • Dependency policy: deny.toml rejects wildcard dependencies and unknown registries or Git sources. Internal path dependencies must include a version. New licenses must be reviewed before adding them to the allowlist.

For detailed style guidelines, see docs/style-guide.md.

Repository Structure

crates/
  infrastore-core/    # Types, HDF5 + SQLite storage, hashing, public Rust API
    src/types/               #   array.rs (TypedArray/Dtype), key.rs, metadata.rs, period.rs,
                             #   time_series.rs
    src/storage/             #   memory.rs, hdf5.rs (storage backends)
    src/metadata/            #   schema.rs (SQLite catalog schema)
    src/store.rs             #   Store: the top-level public API
    src/reader.rs            #   StaticReader / ForecastReader: columnar bulk-read surface
    src/hash.rs              #   SHA-256 column hashing
  infrastore-proto/   # Protobuf service definition (proto/) + tonic codegen, conversions
  infrastore-server/  # gRPC server binary (src/bin/server.rs) + Rust client
  infrastore-py/      # PyO3 bindings
  infrastore-ffi/     # C ABI cdylib (used by the Julia binding)
  infrastore-cli/     # `infrastore` CLI: CSV add/read against an on-disk store (clap, csv, tabled)
    src/chart/               #   hand-written sparkline + SVG renderer (no charting dependency)
    src/commands/            #   one module per command group
  infrastore-bench/   # `infrastore-bench` binary: bulk-ingest + simulation-read benchmarks
julia/InfraStore.jl/    # Julia package wrapping the C ABI
python/tests/                # pytest suite
examples/                    # Sample server config, cli/ (sample CSV + descriptor), and
                             #   runnable python/ + julia/ example programs
.github/workflows/           # Cross-platform tests, linting, security, wheel builds

Build & Test

cargo build --workspace --all-features
cargo test --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings

Prerequisites

HDF5 and zlib are built from vendored sources and linked statically by default, via the vendored feature that every crate enables (defined in infrastore-core, forwarded by the rest). The build therefore needs cmake and a C compiler rather than a system HDF5, plus protobuf for the gRPC codegen:

brew install cmake protobuf maturin                 # macOS
sudo apt-get install cmake protobuf-compiler        # Linux (Debian/Ubuntu)

The first build compiles HDF5 from source (a few minutes), then caches the result.

--no-default-features switches back to system libraries, which then need brew install hdf5 / sudo apt-get install libhdf5-dev, and possibly HDF5_DIR if the hdf5-metno-sys build script cannot locate HDF5. Because hdf5-metno-sys declares links = "hdf5", Cargo's feature unification makes vendored-vs-system all-or-nothing across the whole dependency graph — an individual crate cannot choose independently.

Note that --all-features implies vendored. The workspace dependencies on infrastore-core and infrastore-proto set default-features = false in the root manifest, because a workspace member cannot override an inherited dependency's default-features; each member re-enables vendoring through its own vendored feature.

CI provisions no native libraries on any platform, Windows included — the vendored build covers all three. Do not add a step that exports HDF5_DIR in CI: it redirects the vendored build at an external HDF5 while static libraries are still requested, which fails. Keep these requirements in mind when changing native dependencies.

The workspace cargo config (.cargo/config.toml) sets macOS linker flags so cargo build --workspace can link the PyO3 cdylib without maturin. On Linux and Windows those flags are inert.

Bindings

Python (PyO3)

python3 -m venv .venv && source .venv/bin/activate
pip install maturin pytest numpy tzdata  # tzdata: zoneinfo on Windows
maturin develop --manifest-path crates/infrastore-py/Cargo.toml
pytest python/tests

Julia (C ABI)

cargo build -p infrastore-ffi --release
export INFRASTORE_LIB=$PWD/target/release/libinfrastore_ffi.dylib  # .so on Linux
julia --project=julia/InfraStore.jl -e 'using Pkg; Pkg.instantiate()'
julia --project=julia/InfraStore.jl julia/InfraStore.jl/test/runtests.jl
# The ZonedDateTime tests need the TimeZones weak dependency, which is only
# loadable through the test target; the run above skips them with a warning:
julia --project=julia/InfraStore.jl -e 'using Pkg; Pkg.test()'

julia/InfraStore.jl reads a bare Dates.DateTime as a wall clock (Julia's carries no zone), recording ZonelessReference() — the stored instant is its own fields, unchanged from the old UTC-by-convention reading, but the store now records that it was a convention. It also accepts a TimeZones.ZonedDateTime anywhere a timestamp goes, converting it to the instant it names and recording the spelling its zone names. TimeZones is a weak dependency: the conversion methods live in ext/InfraStoreTimeZonesExt.jl and load with using TimeZones. Reads still return a DateTime holding the instant, with the reference beside it — changing that would break IS3.jl, which destructures them; zoned_timestamp in the extension fuses the two back together.

The FFI build script generates crates/infrastore-ffi/include/infrastore.h via cbindgen. Never hand-edit the header. Any change to an exported extern "C" function must:

  • include an accurate Rustdoc # Safety section covering pointer validity, ownership, lengths, concurrency, and the matching deallocator;
  • regenerate and commit the header;
  • update the Julia wrapper and tests when the ABI behavior changes.

Server

cp examples/server.toml my_server.toml
# edit my_server.toml: point [data].files at your .h5, set [authentication]
cargo run -p infrastore-server -- --config my_server.toml

auth = "api_key" requires at least one entry in keys; clients must send the chosen key in the x-api-key header.

Storage Format

  • A persisted store is an HDF5 file plus a SQLite catalog at <store-path>.sqlite. They are one logical artifact and must be moved, copied, and deleted together. The file is written directly against libhdf5 (via hdf5-metno), not through netcdf-c; the extension is conventionally .h5 but nothing enforces it. Identity comes from the root attribute storage_backend = "hdf5", and Store::open rejects a file that lacks it — including stores written by the removed netcdf backend.
  • CatalogMode decides where the catalog lives while a store is open, independently of the backend. Attached (default) makes it the .sqlite file, with WAL and durability on every commit. InMemory holds it in RAM and writes it only at persist_to or persist_catalog; arrays still stream to the HDF5 file, so it does not require the data to fit in memory. It exists for a consumer building a store in a scratch directory beside its own volatile state (infrasys does exactly this), where a crash loses that state anyway. MemoryBackend + Attached is rejected. persist_catalog writes only the .sqlite half, stamped to match the HDF5 file already beside it — the cheap way to land an in-memory catalog when the arrays are already in place (persist_to to another path has to write them again). The CLI calls it at the end of every add/init, because one command per process means a catalog still in RAM at exit is lost, not deferred.
  • The two halves carry a matching generation stamp — the HDF5 root attribute catalog_generation and the catalog's catalog_identity table. persist_to stages both halves, fsyncs, and renames them into place; because two renames cannot be atomic together, a fresh stamp per save makes an interrupted save fail loudly on the next open (MismatchedArtifact) instead of reading as a valid store. A failed save may still have destroyed the destination — retry from the live store rather than assuming the target survived. compact rewrites only the HDF5 half and must therefore preserve the existing stamp, never mint one. Both halves unstamped (an artifact predating the stamp) still opens; exactly one stamped half is a MismatchedArtifact, because every path that writes a stamp writes both together. Each save stages through a uniquely tagged sibling (<target>.persist-<tag>, <store>.h5.repack-<tag>) — nothing locks a persist_to destination, so a fixed name would let two savers publish each other's partial files. The cost is that an interrupted save's temps are no longer swept by the next one.
  • Creating a store where one already exists is refused (TimeSeriesError::StoreExists), checking both halves. Creating truncates the HDF5 file but only opens the catalog, then stamps both to match, so without the guard a re-run of a build script produced an empty array file paired with the old catalog's rows — opens cleanly, lists every series, every array dangling. create_replacing (overwrite=True / overwrite=true in the bindings) is the explicit destructive form. Store::open_copy copies both halves and opens the copy, so a consumer that means to change a user's artifact never attaches to it read-write; HDF5 has no journal, so an interrupted in-place write is unrecoverable. Both shipped consumers already do this by hand.
  • Timestamps are millisecond-precision. A Period has always been a whole number of milliseconds; every instant the store records (a SingleTimeSeries or forecast initial_timestamp, every entry of a NonSequentialTimeSeries vector, every breakpoint of a PersistentTimeSeries) is held to the same floor, enforced on the write path in Store's validate_data and refused with InvalidParameter rather than truncated. A leap second is refused by the same rule: unix milliseconds cannot express one, so storing it would fold it onto the following second and make two distinct instants one. The reason is cross-binding: the C ABI and Julia exchange instants as i64 Unix milliseconds and Python's datetime is microsecond, so a finer instant is silently truncated at some boundaries and not others. Reads stay permissive so a pre-rule artifact still reads back exactly, which is why the rule does not bump DATA_FORMAT_VERSION. Query bounds (time_range, a reader's when) are deliberately unconstrained.
  • DATA_FORMAT_VERSION in crates/infrastore-core/src/version.rs is the on-disk compatibility contract, checked in three tiers (Current / Upgradable / Incompatible), not by equality. Any incompatible HDF5 layout, dtype encoding, timestamp encoding, or hashing change must bump it, raise MIN_UPGRADABLE_VERSION to match, and update format documentation and compatibility tests.
  • CATALOG_SCHEMA_REVISION (crates/infrastore-core/src/metadata/migrate.rs) is the SQLite half's own contract. Any catalog change the idempotent DDL cannot make to an existing table — a new column, a changed CHECK, a rebuilt table, a backfill — now requires a CATALOG_SCHEMA_REVISION bump plus an append-only MIGRATIONS entry, not a re-created store. Never edit a landed migration; add a new one, and give it a frozen snapshot of the shape it produces rather than deriving it from the live DDL. A writable open climbs the ladder before re-applying the DDL and then re-stamps the HDF5 half; a read-only open of a stale catalog reports CatalogMigrationRequired (the actionable error: open it once for writing), and a catalog from a newer build is CatalogTooNew. A purely additive new table or index still needs neither bump.
  • Packed arrays use datasets named sts_{dtype}_{shape}_{length}_{resolution} for regular series and nsts_{dtype}_{shape}_{length}_{timestamps_hash} for the irregular ones sharing a time axis, each with a companion <dataset>_h hash dataset. Standalone arrays use arr_{hex_hash}. A NonSequentialTimeSeries's timestamps — and a PersistentTimeSeries's breakpoints — live in the HDF5 file too, as one tsv_{hex_hash} i64 dataset of unix milliseconds per distinct time axis under time_series/timestamps/, keyed by the same content hash that pools their arrays. See crates/infrastore-core/src/storage/hdf5.rs for the implementation and docs/src/reference/file-format.md for the user-facing specification; keep them synchronized.
  • Deletion frees a packed column (slot reusable, hash row and column data zero-filled) or unlinks a standalone dataset. HDF5 cannot return the space in place, so the file only shrinks when Store::compact rewrites it: an on-disk compaction materializes the catalog's live arrays into a sibling <store>.h5.repack and renames it over the original, assuming a single writer. Compaction behavior must remain explicit.

Conventions

  • Keep the multi-language surface consistent: a change to the core public API usually needs matching updates across the proto definitions, the gRPC server/client, the PyO3 bindings, and the FFI/Julia binding. When adding a feature, check all bindings before considering it done.
  • Treat infrastore-core as the source of truth. Binding crates depend on core; core must not depend on bindings.
  • Use TimeSeriesError and the shared Result alias for core errors. Unsupported operations must return an explicit error rather than silently changing semantics.
  • Preserve typed-array dtype, shape, byte order, timestamps, features, and hashes across every binding and persistence round trip.
  • Do not manually edit generated artifacts. Besides the C header, protobuf output is generated by the proto crate's build script.
  • Keep changes scoped. Do not commit local virtual environments, Python caches, generated HDF5 test data, or machine-specific library paths.