Prefixes for extensions

[m-mohr]

From the fiboa Slack:

@cholmes wrote:

I'm working on a Planet converter (got to a PR that needs some help) and started on an extension for Planet's extra values, to try to do it a bit more 'proper'. I did them as planet:qa and planet:mcid, but I'm questioning if we should bring our stac convention of the : with a prefix over or not. It doesn't work great in SQL (you have to quote it), and I feel like it's a special character in other places too. I'm sorta contemplating just not doing (as much?) prefixing. Like it'll be easier to have Planet's geopackage (that's not on fiboa) have an equivalent geoparquet/fiboa that has mostly the same fields, especially for the ones where we don't have a 'standard'. And then also wondering about just doing like tillage_occurred instead of tillage:occurred for places where a prefix does make some sense.

@m-mohr wrote:

The : is indeed not ideal, but on the other hand _ is too generic and may lead to conflicts with existing fields much more often. Also, the _ might be ambiguous... crop_type_identifier_code - What's the prefix? crop? crop_type? What if someone defines an extension crop with a field type_identifier and an extension crop_type with a field identifier?

@andyjenkinson wrote:

I think the colon has a clear place as a URI prefix. However one thing we could do is to actually make the spec JSON-LD. That means the schema properties are full URIs, and an extension is just a URI prefix (namespace) for the properties within that extension. Then you use a JSON-LD context object (which can also be shifted to an external URL) that maps property keys to a full URI. That makes your JSON look like normal JSON (and can even be a way of keeping all the proprietary property keys you already have) but automatically convertible to the common standard. Basically you could largely retrofit FIBOA extension compatibility in an almost invisible way. By the way it also means you can literally reuse concepts that already exist in other ontologies, like DCAT for datasets, PROV-O, agri domain ontologies etc.
To me, I think a great target would to aim that an implementation can hit FIBOA, JSON-LD and OGC Features API compatibility all at the same time. The actual spec for an extension would be expressed in JSON-LD, but all of the examples would look like plain old JSON with one ‘@context’ property that contains the schema mapping.

[cholmes]

Also, the _ might be ambiguous... crop_type_identifier_code - What's the prefix? crop? crop_type? What if someone defines an extension crop with a field type_identifier and an extension crop_type with a field identifier?

Yeah, I think the essence of the idea to me is more to stop worrying about prefixes. Like letting Planet do 'mcid' instead of 'planet:mcid' or 'planet_mcid'. This may just be a bad idea - but I'm just sorta wondering what we've really gained by having all the prefixes. If you have a data model and want to validate it with a few different extensions then you'll be making choices about what to validate it with. The chances of overlap seem small, and if there's a set of 'known' extensions then people introducing new extensions that might need to be compatible can tweak their names. It would essentially just be a 'looser' approach to the ecosystem - here's a set of attributes that mean these things, but they're not trying to define things for all times.

So like Planet would have 'mcid' defined at https://planet.github.io/fiboa/planet-fb-extension/v0.1.0/schema.yaml, some other org can have 'mcid' (and maybe it means something different...) defined at https://company.com/fiboa/company-extension/v0.1.0/schema.yaml, but if they wanted to make the ecosystem more compatible then they could just establish a new community extension at https://fiboa.org/mcid-extension/v1.0.0/schema.yaml, but the field name would stay 'mcid' - it'd just use the community-build JSON schema to validate.

I think the colon has a clear place as a URI prefix. However one thing we could do is to actually make the spec JSON-LD. That means the schema properties are full URIs, and an extension is just a URI prefix (namespace) for the properties within that extension.

And yeah, I think this is the other extreme of the approach, attempting to have the prefix have 'real' meaning. The original idea of the prefix in STAC was inpired by JSON-LD, with the intent to try to do just what you're saying @andyjenkinson - tie the prefixes to full URI's with well-known meaning in JSON-LD. I think one thing that threw it off is that the 'geo' representation in JSON-LD wasn't great if I remember right, and very few tools had support for it. But it's probably worth taking another run at figuring out if we could fully support it - I agree that hitting 'FIBOA, JSON-LD and OGC Features API compatibility all at the same time' would be really great.

The actual spec for an extension would be expressed in JSON-LD, but all of the examples would look like plain old JSON with one ‘@context’

But for extensions wouldn't you need to one 'context' per extension? Unless you put all the extensions into a single 'fiboa' context? Like if you don't put them all in a single context then there'd still be prefixes for all the ones that aren't the primary / default context?

It also seems like when you map from JSON-LD to GeoParquet you'd need to bring the prefixes back in consistently, or else try to fully represent the URI's in Parquet.

[andyjenkinson]

The context is a property included at the root of each payload, the value of which is either a context object or a URL of one, rather like a hypermedia link. So it can be unique to each implementation/dataset, and can include terms from any number of extensions. The extensions would give example contexts that correspond to example payloads, but when you implement FIBOA you can either:

• merge the contexts from the extensions you want verbatim into your own context file, and then name all your properties like in the examples
• rename some properties in your context to tailor your format to the property names you want (particularly useful if you have an existing implementation you don’t want to change)

Bear in mind JSON-LD contexts can remap any property to a URI, not only expand a prefix. This would allow Planet to have whatever terms it wanted in its payloads, they wouldn't even have to match the name of the property in the FIBOA spec and don't have to contain any prefixes; the mapping to FIBOA would be done entirely by the context file. All the machine readable stuff like validation, conversion etc can use the processed JSON-LD representation, but the files look completely normal to users and 'FIBOA unaware' software.

Regarding parquet, yes either you define them as full URIs, or you would carry forward the JSON-LD context mappings into the headers (and vice versa of course) so that they look ‘normal’ in any existing software that processes parquet data. Basically the headers would have to say "these are the property names, and these are their equivalent URIs.

The one thing I’m unsure about is the geo stuff. You may know there is a GeoJSON-LD context but I have not looked at it in detail. I would not want to abandon GeoJSON for some other random representation of geometry in JSON-LD, it’s about making a standard GeoJSON payload processable as JSON-LD. In fact that context is a good example explaining what I mean above about using the context object to essentially make JSON-LD look like completely normal JSON unchanged from its original format. All it is is a context, which maps all the original GeoJSON schema items to URIs exactly as they are.

Personally I am not a fan of going the other end of the scale and just allowing a free for all on names. I get namespaces are annoying but I can see clashes happening especially for terms like "crop", and in particular it's useful to distinguish 'uncontrolled' terms - personally I'm not sure it's necessary to make a Planet extension as by definition there won't be any terms in common with anyone else. So long as FIBOA allows additional properties just document your schema and anything proprietary doesn't need a prefix. Then focus on trying to standardise things that seem common in a topic- not vendor-specific extension. Unless you adopt JSON-LD or something like it, someone's going to have to change their schema anyway.

[cholmes]

@m-mohr - why didn't you use a prefix on flik-extension?

[m-mohr]

Good question, it was my first one and more an example. The fields in the original had no prefix, I guess I either forgot it or thought it's simpler, can't remember 😅

[cholmes]

Cool. Yeah, both reasons to me point a bit to how it could be nicer to not have to think about them.

Curious what you think about JSON-LD and contexts, and if you'd be up to dig into it a bit. Like if there is a way to enable us to pass through the 'schema' information without including the prefix, like all the way through geoparquet. I'm a bit less sure how much GeoParquet metadata should really handle - I wonder if there's any other examples of JSON-LD -> Parquet. And if it'll work when pulling a few different 'contexts' into one.

[m-mohr]

I'm travelling next week, but I can dig into it afterwards, but it will likely take a three weeks or so. It doesn't seem to solve the colon / quote issue though. I'm not sure whether we can solve that if the allowed set of characters for SQL names is A-Z, 0-9 and _.

I worry a bit that without a prefixes we'll end up with various extension that use crop_id and maybe even datasets that use no extension and have crop_id. If you want to merge them, what do you do with the fields names? You can't do it because the field is differently defined.

In STAC we see that many clients actually don't check stac_extensions array and just use the fields because they can be sure there are usually no conflicts. Clients would need to be developed with more care, they may even need to read all schemas.

So I tend towards a prefix at least for "common"/sebatable field names. If you have names like "flik" that are unlikely to conflict I could see that we allow without prefixes. We also sometimes do that in STAC.

JSON-LD is an open question for now.

[andyjenkinson]

It could solve the colon issue because there won’t be any colons in the payload any more, only in the context file which only ever needs to be read when doing things like validating or converting. And they’ll be in values, not keys. You can put whatever property names you like in your implementation, they don’t have to be named the same as the ‘standard’ ones, the context file provides a mapping. So the GeoJSON file literally looks like any normal JSON with one extra property ‘@context’ that links to the context. Everything else can be your native implementation, and if you want that to be a translation of a SQL schema, have at it. Think of the context as a set of instructions of how to convert the GeoJSON Feature/FeatureCollection to a FIBOA JSON-LD schema. It’s pretty much a glorified ‘find and replace’.

So for example in your JSON you could have:

{
 '@context': 'https://planet.com/path/to/my/context.json',
 'id': 'ABCD1334',
 'date': '2024-06-29'
}

And after mapping if would look something like:

{
 'http://purl.org/dc/terms/identifier': 'ABCD1234',
 'https://fiboa.org/schema/core#determination_date': '2024-06-29'
}

Here, I use an example where the FIBOA core specification would define the 'id' property, as a mapping to the existing Dublin core vocabulary term 'identifier', as well as its own unique properties.

Meanwhile the FIBOA examples (and the format you'd use if you were creating a file from scratch) could look like:

{
 '@context': 'https://fiboa.org/schema/context.json',
 'id': 'ABCD1234',
 'determination_date': '2024-06-29',
 'myextension:image_resolution': '10m'
}

And that would map to an identical RDF graph as the Planet example:

{
 'http://purl.org/dc/terms/identifier': 'ABCD1234',
 'https://fiboa.org/schema/core#determination_date': '2024-06-29',
 'https://fiboa.org/schema/myextension#image_resolution': '10m',
}

I've simplified the structure of all these of course. I'm typing on my phone.

The standard context file would contain all the mappings from the core and extensions. Here the only reason for the colon is for the same reason we have it today: to allow independent development of extensions which might simultaneously use the same property name. But if this isn't important (eg each extension is allowed to basically claim a property name by using it first) it could be removed. Either way, each extension would have to provide a standard JSON-LD context that would translate the "nice plain JSON" to full messy URIs. The key to this is to hide as much as possible the mechanics of JSON-LD to make creating and maintaining extensions and making compatible features as easy as possible.

[m-mohr]

Okay, I'm currently trying to wrap my head around this. I might have misunderstood parts of JSON-LD, please let me know if that's the case.

Note

Our extension mechanism doesn't require by any means a colon, it's a (pretty undocumented) best practice.

We are used to it through STAC and it's primarily used to easily distinguish the fields while not looking at the list of implemented extensions. All fields can be unambiguously identified through their name + extension URI, but many STAC readers (except for the validators) actually do not use the stac_extensions to actually verify which extension fields belong to. They simply use the field name as it is and assume that field names are unique. If someone would create a new extension with overlapping names of an existing well-known extension (e.g. an eo2 extension with eo:cloud_cover from 0 - 1), most STAC implementation would not differentiate it and assume it's a percentage from 0-100.

JSON-LD vs fiboa extensions

My current understanding of JSON LD and our current extension mechanism in fiboa is that they are very similar conceptually.
JSON-LD is a bit more advanced, but is also more complex to implement.
The fields @contexts and fiboa_extensions are pretty much the same, they define through URIs how certain fields are named and their corresponding semantics. Through that the fields can be identified unambiguously, if clients actually make use of the URIs.

This seems to be rarely the case (in STAC and fiboa) because it makes implementations more complex. While we can implement reference implementations that do that, user land implementations have proven through STAC that people take the simplest route and don't actually check against the provided extensions whether their assumptions about a field are actually true.

Let's say we have three extensions that all define a crop field, but one of the extensions is very dominant and popular. Tools can distinguish the fields if implemented properly. But people would usually just address it through the crop name and not always check whether it's actually the right extension. With crop you may figure it out due to unexpected values, but the cloud cover example above is not so obvious.

For JSON there is LD tooling that could probably mitigate this, but for GeoParquet there's not. The question is whether people would use such tooling or whether they just use their normal JSON or Parquet reader. In this case they won't read the URIs (neither context nor fiboa extensions) and then you run into potential issues. For Parquet, you could prefix all fields by URI, e.g. https://planet.con/ext/crop.yml:crop), but that looks even worse than planet:crop.

Prefixes

I think JSON-LD (or fiboa extensions) without prefixes looks nice, but has quite a number of hurdles.
While quoting fields with prefixes is an annoyance, it's likely nothing that actually affects the analysis and people are used to this, I think.
On the other hand, the crop or cloud cover examples could actually negatively affect the analysis when conflicts are not properly resolved.
I feel like we should provide a framework where people can work with easily and that prevents error. As such I'd keep the prefixes for fields where conflicts are likely to happen. Having that said, you can actually derive from that. I don't expect conflicts for flik and as such having no extension for the sake of keeping the files as close to the source as possible is an option.

Merging files

The "merge" issue that I spoke about in another comment before, can be resolved by implementing the merge tool in a way that if conflicts arise, the full URI will be used as a prefix. So for example the following two files would be merged as follows;:

• File 1 (columns: id, geometry, crop [from https://example.com/crop.yml])
• File 2 (columns: id, geometry crop [from https://planet.con/ext/crop.yml])

Merged file has columns: id, geometry, https://example.com/crop.yml:crop, https://planet.con/ext/crop.yml:crop)

Renaming (for compatibility)

The "rename" machanism in JSON-LD looks good on first hand as it allows to use fiboa without changing the actual field names in an existing implementation, e.g. OGC API - Features. Users need good clients or a good understanding of LD to make the connection though.

What about the values? If you need to change the values, e.g. from area in meters to area in hectares, how do you express in JSON-LD that you need to divide by a certain value? I couldn't find anything about it, so I assume that's not a thing. So the rename covers just a small part of being compatible with an existing response and as such I feel like it's not worth the hassle.

Compatibility

The compatibility between fiboa, JSON-LD and OGC API - Features would indeed be nice.
I don't quite see issues with OGC API compatibility at this point though. It might conflict to specific implementations and content schemes that providers offer, but fiboa is not incompatible to OGC API - Features afaik.

The field renaming could solve that partially, but I think in many cases it would just cover parts of it (see the area example above). So you may still not be compatible with specific implementations and content schemes although you renamed everything nicely.

So it leaves us with JSON-LD vs fiboa. I'm not quite decided on this yet. I generally find it hard to navigate the JSON-LD vocabularies. Like how can I verify whether there is an field boundary related vocabulary? There's GeoJSON-LD, but is there more? I'm not sure whether compatibility with JSON-LD would help us a lot. Is anyone asking for it? The fiboa extension mechanism is very similar. We'd need to define JSON-LD vocabularies in addition to the relatively simple fiboa Schemas, not sure whether they could be generated from the fiboa Schemas.

We could probably go for JSON-LD if we think it's wort it, but currently, I'm not sure whether it's worth the effort. It might be worth the effort if we do more then field boundaries in the future, but I feel like field boundaries wouldn't benefit a lot from it unless we find existing LD vocabularies for it.