Right now things just write to local JSON or to BigQuery; it would be desirable for prod to keep a copy of the JSON on GCS as well as write to BigQuery. Ideally it'd be zstd'd at a certain size and copied over.
As for whether it should be pre- or post- hydration JSON (or both), I'm fine either way.
Right now things just write to local JSON or to BigQuery; it would be desirable for prod to keep a copy of the JSON on GCS as well as write to BigQuery. Ideally it'd be zstd'd at a certain size and copied over.
As for whether it should be pre- or post- hydration JSON (or both), I'm fine either way.