Support a proper xarray backend

[sjperkins]

Currently, xarray-fits defaults to construction of dask arrays which are inserted into xarray Datasets.

More flexibility could be attained by implementing a formal xarray backend

[caseyjlaw]

I came across this issue while researching solutions for xarray in radio astronomy. Our team that will be developing xarray support for OVRO-LWA data.
Our data is assumed to be in zarr format and stored in the cloud. Ideally, we would also support local zarr and fits files, too. Given a path or URL, we can then have an xarray dataset that can be manipulated via averaging, selection, etc.
Is that a use case that you would like to be supported? If so, we could coordinate with you.

[sjperkins]

I came across this issue while researching solutions for xarray in radio astronomy. Our team that will be developing xarray support for OVRO-LWA data. Our data is assumed to be in zarr format and stored in the cloud. Ideally, we would also support local zarr and fits files, too. Given a path or URL, we can then have an xarray dataset that can be manipulated via averaging, selection, etc. Is that a use case that you would like to be supported? If so, we could coordinate with you.

@caseyjlaw I think that would be very helpful. Indeed, the reason for this issue was due to the success of implementing an xarray backend in xarray-ms: https://xarray-ms.readthedocs.io/en/latest/introduction.html#why-xarray-ms.

Just to clarify, are you targeting support for FITS, or other legacy formats too?

With regard to FITS, it would be great if the resulting xarray view of the data could be organised in an xarray DataTree. I imagine a node per HDU would work well, although there may be other ways of organising the data.

Also, are you envisaging opening FITS files through the xarray.open_datatree function (i.e. using patterns similar to here)? I would encourage this as it abstracts the underlying format away from the software developer (who just needs to use xarray constructs) and will hopefully make the transition from legacy (FITS) to modern (Zarr) formats far simpler.

[o-smirnov]

This would be a good context to revisit the "cube dataset" format we adopted in TRON and ideally, align it with whatever @caseyjlaw has in mind...

[sjperkins]

This would be a good context to revisit the "cube dataset" format we adopted in TRON and ideally, align it with whatever @caseyjlaw has in mind...

Indeed, I don't think we've ever found the time to sit down and formalise the xarray coordinates and attributes for FITS-like data.

[caseyjlaw]

Thanks for the comments; this is encouraging.
We are focused on FITS support, since our pipelines create FITS files. We have not thought about the actual structure of the dataset, so would defer to the expertise and the development demonstrated in this repo!
I'm not familiar with plans for MSv4. Is that meant to be a standard that allows zarr format MS files that contain visibilities or images? If so, is it expected that some imaging software (e.g., CASA, wsclean) will support writing zarr and FITS?
Part of our team includes software developers at Univ of Washington that have worked on the xarray core. It may help to arrange a chat that includes all of us, at least to get the requirements and development plans clear. Would you be open to that?

[caseyjlaw]

One other question: has anyone looked into data weights in xarray? A requirement for our team is to support optimal co-addition of images. There are a few ways to implement that, but in optical contexts, it relies on a per-pixel weight.

[sjperkins]

Thanks for the comments; this is encouraging. We are focused on FITS support, since our pipelines create FITS files. We have not thought about the actual structure of the dataset, so would defer to the expertise and the development demonstrated in this repo! I'm not familiar with plans for MSv4. Is that meant to be a standard that allows zarr format MS files that contain visibilities or images?

The MSv4 spec doesn't specifically mandate a format, but for all intents and purposes the format will be zarr.

If so, is it expected that some imaging software (e.g., CASA, wsclean) will support writing zarr and FITS?

I think they're intending to move towards zarr pipelines. Certainly for distributed applications, zarr makes sense because one can write multiple chunks in parallel. That's more tricky with legacy formats. I think efforts are underway to define a spec for imaging data products but I've been more focused on the visibility side of things.

Part of our team includes software developers at Univ of Washington that have worked on the xarray core. It may help to arrange a chat that includes all of us, at least to get the requirements and development plans clear. Would you be open to that?

Yes, definitely keen to have a chat!

One other question: has anyone looked into data weights in xarray? A requirement for our team is to support optimal co-addition of images. There are a few ways to implement that, but in optical contexts, it relies on a per-pixel weight.

I haven't, but @o-smirnov or @landmanbester may know more

[landmanbester]

One other question: has anyone looked into data weights in xarray? A requirement for our team is to support optimal co-addition of images. There are a few ways to implement that, but in optical contexts, it relies on a per-pixel weight.

I'm not sure I understand what you mean by data weights or optimal co-addition of images (sorry, not an optical astronomer). Do you mean a spatially weighted sum as is required for mosaicking? I don't think there is a built in method to do that but I can see it being quite useful.

If so, is it expected that some imaging software (e.g., CASA, wsclean) will support writing zarr and FITS?

I think this is definitely on the cards for ngCASA (or whatever it will eventually be called). We put out a paper series describing our efforts (see Africanus I and references therein) to develop more data source agnostic pipelines for radio interferometry late last year. I don't think it is too difficult to wrangle FITS into xarray + zarr but I'm not aware of any viewers that can cope with that at present. A proper xarray backend for FITS might go a long way towards supporting that.

Part of our team includes software developers at Univ of Washington that have worked on the xarray core. It may help to arrange a chat that includes all of us, at least to get the requirements and development plans clear. Would you be open to that?

Definitely!

[caseyjlaw]

Excellent! I think the xarray core devs will be excited to know that radio astronomy is moving in this direction.
Our team meets weekly on Thursdays at 1pm PT. I'm guessing that is too late for you. Shall I post scheduling poll here for a chat next week?

[caseyjlaw]

For a bit of context, here are some notes from our collaboration's last meeting: uw-ssec/ovro-lwa-portal#6.
The broader goal of the collaboration is to support visualization and light analysis of a 96 TB image cube from the OVRO-LWA (all sky imaging from 18-85 MHz). The cube is composed of FITS images (RA, Dec) at different (frequency, time, polarization), so as a zarr it may be set up as a 5D distributed array.

[o-smirnov]

The broader goal of the collaboration is to support visualization and light analysis of a 96 TB image cube from the OVRO-LWA (all sky imaging from 18-85 MHz). The cube is composed of FITS images (RA, Dec) at different (frequency, time, polarization), so as a zarr it may be set up as a 5D distributed array.

I think this could be straightforward (famous last words, I know...) We've been talking to the CARTA people about providing native support for our chunked zarr format, since we also generate 5D multi-TB cubes in our transients pipeline. It seems they have the machinery in place for visualizing chunked cubes already, so it's primarily a formats and interfaces problem at this point. Your use case could provide the additional impetus that this collaboration needs -- let's talk to them!

[caseyjlaw]

Great. Kumar Golap is also collaborating with us and connected with developments on the NRAO side. Seems like this idea has legs!
The software devs will be free to chat on Tuesday, Sept 23 and in the UK time zone. They'll reach out soon, I believe.

[caseyjlaw]

I am also in contact with Jan-Willem at NRAO about xradio. First step is to test parsing of our data and verify that metadata look right. It doesn't sound like there are significant new requirements for using xarray/zarr for OVRO-LWA (and DSA-2000) data.