How does one query telstate?

[sjperkins]

I've been digging through telstate and katdal to try and figure this out.

AFAICT katdal sets up a sensor cache and usually extracts scan information from self.sensors.get("Antennas/array/activity") or perhaps self.sensors.get(f"Antennas/{ref_ant}/activity") if there is a reference antenna. The resultant scans have the expected labels such a "slew", "track" etc.

I can't figure this out from the raw telstate though. I've looked in places like "obs_label", "obs_activity", "mxxx_activity", but the activity seem to just have "stop" in them. There seems to be an expansion occurring in SensorCache using data from locations I don't know about.

@ludwigschwardt can you give insight here?

I guess I could cut n paste the katdal SensorCache code into xarray-kat, not sure if this is necessary or overkill.

[ludwigschwardt]

Of course the main question is: why not just depend on katdal? Too heavy / dasky? 😂

I see katdal as "telstate++" combined with chunking. At least half its utility is deciphering telstate... These sensors are unfortunately not trivial to parse (see e.g. the longstanding SPP1-108).

I could consider splitting that out to a "katdal-lite" (or a restricted dependency set) or (as @bmerry would like) write the decoded RDB to yet another metadata database format. 🤔

[sjperkins]

Of course the main question is: why not just depend on katdal? Too heavy / dasky? 😂

Yep, in a nutshell, being defensive vs dependencies (I suspect we've all learnt the hard way). Ofc, I'm making bets on xarray and tensorstore here, maybe a critical look on these ideas are needed, but also nothing is certain and we need a bias towards action.

I see katdal as "telstate++" combined with chunking. At least half its utility is deciphering telstate... These sensors are unfortunately not trivial to parse (see e.g. the longstanding SPP1-108).

Yeah I'm starting to get a feel for the issue: State is dumped into a redis server during an observation and the dumped data is not necessarily deterministic or well-behaved? These discrepancies then need to be resolved and presented via the dataset interface somehow?

I could consider splitting that out to a "katdal-lite" (or a restricted dependency set) or (as @bmerry would like) write the decoded RDB to yet another metadata database format. 🤔

I actually started vendoring some katdal code into xarray-kat today just to start thinking about this, but didn't get very far at all: Idea is be to cut anything chunkstore and dask related but leave the rest intact.

I do like the idea of a katdal-lite which would only concern itself with interpreting sensors. In my ideal world this would only take a python dictionary: it would be the responsibility of the caller to decode from RDB. Then again, I'm thinking in an offline manner, I've seen the sensor code that interacts directly with the redis server.

Were there any other metadata formats that you and @bmerry were considering? I also noticed the rdb dumps getting rather large: off the top of my head something small defining the "scaffolding" of the observation, then something else storing the majority of the sensor data might work? Arrow tables would work well for time series data.

[sjperkins]

then something else storing the majority of the sensor data might work?

https://www.influxdata.com/products/influxdb/ for example.

[ludwigschwardt]

I do like the idea of a katdal-lite which would only concern itself with interpreting sensors. In my ideal world this would only take a python dictionary: it would be the responsibility of the caller to decode from RDB. Then again, I'm thinking in an offline manner, I've seen the sensor code that interacts directly with the redis server.

Yes, our grand scheme is to have katdal run directly on the redis server in our pipelines but this was never really put into action 😅 See SR-1787.

Since bugs are still being fixed on the sensor front, I'd really prefer a central place for the parsing, as opposed to vendored bits and bobs.

Were there any other metadata formats that you and @bmerry were considering?

Bruce liked SQLite.

I also noticed the rdb dumps getting rather large: off the top of my head something small defining the "scaffolding" of the observation, then something else storing the majority of the sensor data might work? Arrow tables would work well for time series data.

I would prefer a single file but redoing the archive is also a bit onerous at this point. My current thinking is to tack on an index to the basic RDB, which should help a lot. All fields are stored in contiguous byte ranges, so getting large chunks directly out of the RDB monolith is quite doable. The small values / attributes could be stored with the index in a dict of sorts.

Another helpful project for this is librdb. I'm just waiting for someone (or me) to write a Python binding 😁

[sjperkins]

All fields are stored in contiguous byte ranges, so getting large chunks directly out of the RDB monolith is quite doable.

Right, so scan the rdb structure, generate an index and using HTTP range requests to pull the relevant bits out.

Another helpful project for this is librdb. I'm just waiting for someone (or me) to write a Python binding 😁

I'll block off some time for a feasibility study with librdb.

[ludwigschwardt]

Right, so scan the rdb structure, generate an index and using HTTP range requests to pull the relevant bits out.

Exactly.

I'm hoping to still support redis natively, so librdb should come in handy. I see you've forged ahead on that front 😅 Just be aware that there are basically just two developers (although it is at least an officially sanctioned product, and already feature complete for our purposes).

[ludwigschwardt]

There is at least one attempt of a Rust route to RDB Python bindings: https://github.com/bimtauer/rdb-rs. It's still unreleased though (and a solo operation :-)).

[sjperkins]

@ludwigschwardt I can see three requirements here:

1. Support newer versions of redis within the SDP pipeline.
2. Interact with newer redis database files in python due to the deprecation of rdtools.
3. Query small parts of the rdb file over the network to (a) establish the outline of a katdal/xarray Dataset and (b) avoid pulling down potentially GBs of sensor data before it needs to be used.

Generally, I think we're seeing the following strategies emerge:

1. Create python bindings for librdb so that katsdptelstate can interact with newer redis database files
2. Identify key-value byte ranges in the rdb file for which it may or may not be possible to decode using individual librdb functions. Although it's possible to provide the RdbParser with a buffer of bytes, this would also probably involve tricking the RdbParser state so that it realises that it's in the middle of a file (and not at the beginning).
3. I'd find it interesting to do the parsing in rust and the python bindings with PyO3, much like @bmerry has done for katcp-codecs.

HOWEVER, there's a solution which would satisfy the first two requirements: Export the RDB file to JSON and consume that in katsdptelstate: librdb's example code contains a JSON exporter which could be run on the dumped rdb file. It's even smaller than the rdb file when compressed:

-rw-rw-r-- 1 simon simon 737M Nov  5 12:00 1743509485_sdp_l0.full.json
-rw-rw-r-- 1 simon simon 164M Nov  6 14:08 1743509485_sdp_l0.full.rdb
-rw-rw-r-- 1 simon simon  89M Nov  6 13:39 1743509485_sdp_l0.full.tar.xz

With regards to the third requirement, a 1743509485_sdp_l0.tiny.json could contain a subset of the full file that could excise most of the sensor data but provide sufficient information for establishing a dataset view over the data.

Probably this is the simplest and quickest route?

[bmerry]

@ludwigschwardt I can see three requirements here:

1. Support newer versions of redis within the SDP pipeline.

2. Interact with newer redis database files  in python due to the deprecation of rdtools.

3. Query small parts of the rdb file over the network to (a) establish the outline of a katdal/xarray Dataset and (b) avoid pulling down potentially GBs of sensor data before it needs to be used.

I'd say it's also a nice-to-have to be able to open an RDB file that's on disk and be able to start querying attributes immediately without having to wait a minute or two to parse the entire file. That's probably easier than (3) since reading small pieces off disk has orders of magnitude less latency than HTTP range requests. Possibly just having a C-based parser instead of the pure-Python rdbtools will speed things up enough to remove the pain, but it may need something a bit smarter.

[sjperkins]

Some notes from this afternoons discussion.

• I thought the rdb was generated from a dump by redis. This is not the case: The rdb is actually hand-crafted from raw Python bytes
• This might provide more support for avoiding using rdb at all: json/msgpack are alternative formats with extensive support not only for Python but across many other languages. They're also simple formats.
• This doesn't necessarily imply rdb should be deprecated immediately: most observational metadata is stored in this format.
• However, @sjperkins suspects it should be possible to ingest a json/msgpack encoded dictionary into katsdptelstate.TelescopeState as a alternative format to rdb.
• But this needs testing and some thought is still needed to determine exactly how to plug the resulting dict into katsdptelstate.TelescopeState.
• @sjperkins went over the flag/weight/vis connections, and needs to investigate how tensorstore's caching mechanism might work with the interplay between these data variables.

[sjperkins]

Another concept w.r.t. metadata: A JSON/MsgPack index could contain the a minimal amount of metadata required to establish a dataset over the underlying chunk data. This is similar to the existing rdblite SDP idea.

However, it would contain the entire set of metadata keys, but most of them could contain indirect links to other JSON/MsgPack files containg the majority data. This would allow telstate to be initialised with minimal metadata. Then, if a key with indirection was requested by a user, the referenced JSON/MsgPack file could be read from disk/downloaded and merged into the telstate.

index = {
  "start_time": ...,
  "ndumps": ...,
  "dump_rate": ...,
  "chan_freq": ...,
  "azel_pointing": LinkMarker("pointing.json"),
  "m03_humidity": LinkMarker("weather.json"),
   ...
}

pointing = {
  "azel_pointing": [[0, 0], [0, 1], [0, 2]],
  ...
}

weather = {
  "m03_humidity": [0, 1, 2, 3, 4],
  ....
}

Current issue with this idea is that a lot of telstate data may be read and converted into sensors upfront. Perhaps more lazy-loading could be built into the telstate/sensor repo referred to in #4 (comment)?

I could consider splitting that out to a "katdal-lite"

[bmerry]

Another concept w.r.t. metadata: A JSON/MsgPack index could contain the a minimal amount of metadata required to establish a dataset over the underlying chunk data. This is similar to the existing rdblite SDP idea.

The trick is making a choice of what's considered "minimal" that keeps everyone happy. That's why I think just using a file format that allows random access (such as sqlite, or possibly parquet although I haven't looked into it) is a better approach, since then every use case can hit as fast a path as possible, rather than only having a fast path for specific cases that the developers thought up.

However, it would contain the entire set of metadata keys, but most of them could contain indirect links to other JSON/MsgPack files containg the majority data. This would allow telstate to be initialised with minimal metadata. Then, if a key with indirection was requested by a user, the referenced JSON/MsgPack file could be read from disk/downloaded and merged into the telstate.

Keep in mind that a not-uncommon use case is "give me the values of all these sensors over this small time window"; for that you want to have random access within each sensor rather than being forced to download a file with the entire history of a sensor.

Do you think there is something to be said for having the archive provide an HTTP endpoint that provides effectively the (read-only) telstate API, and have katsdptelstate support that? That way we can experiment with converting to different formats behind the scenes but not commit to supporting a specific file format in katsdptelstate/katdal that we then have to support forever.