Import Showdown - what imports to where in the ATO?

[jdpye]

A shopping list for code coverage:
we will need to map to/from these package types

Core

• actel
• glatos
• otn detext / parquet
• otn metadata
• yaps
• otndo
• IMOS / remora
• Manufacturer formats
  • Innovasea
  • Lotek
  • Thelma
  • Sonotronics

Stretch

• patter
• atFiltR
• OTN Parquet public via ERDDAP
• OTN metadata via GeoServer

[mhpob]

OBIS/Darwin Core?

[jdpye]

There are a couple of network-level pipelines towards OBIS/DwC - the publication standard requires a summarized data product rather than the raw data itself. We could use the mappings over at https://gitlab.oceantrack.org/otn-partner-nodes/ipython-utilities/ to get the columns mapped to their DwC terms but those archives wouldn't be publishable in raw form.

[jdpye]

now that you've breathed new life into it, what does TelemetrySpace work with / need? Does it belong on this list?

[jdpye]

sporeg is a package that's poised to leverage the ATO centralization - and you and Beth as its maintainers are engaged and potentially it could be a good pilot integration that identifies the documentation needs and scaffolding that new package authors might need from us as well as being Another Endpoint To Reach

[mhpob]

now that you've breathed new life into it, what does TelemetrySpace work with / need? Does it belong on this list?

I'm going to kick this to @benjaminhlina as he has a much better idea on this.

sporeg ... could be a good pilot integration that identifies the documentation needs and scaffolding that new package authors might need from us as well as being Another Endpoint To Reach

Once the funder issues the funds, Beth and I will poke around at this.

[benjaminhlina]

I would say yes it does belong on this list if we are wanting this to talk to analysis packages and not just data manipulation. TelemetrySpace (we may do a migration of such; and renaming) requires a summary array with dimensions of n_tag, n_time, n_det x station.

My idea right now which I have done locally is to have the functions of telemetryspace likely be given a summary dataframe generated from a detection extract/ATO and then internally it makes all of the objects that Stan needs for instance the summary array.

[benjaminhlina]

So yes it would be nice to have a standard schema (ATO) to rely on to build this out.

[jdpye]

I agree Ben, that's a nicer place to draw the line, as well, if analysis requires a specific summary or subsetting or nudging of the source data, best to let that happen in the analysis package, from the core standard format. That way if the analysis package designer changes the requirements they are in control of the methods that reorganize the data they're ingesting.

[benjaminhlina]

Correct!