Standardize [Packed]State

[Affie]

TABLE IS STILL A WIP (Starting discussion on val)

	State	Old PackedState	New PackedState Field
✓	initialized::Bool	initialized::Bool	initialized::Bool
✓	× id::Union{UUID, Nothing}	× id::Union{UUID, Nothing}	REMOVE
✓	× dimIDs::Vector{Int}	× dimIDs::Vector{Int}	REMOVE
✓	× solveInProgress::Int	× solveInProgress::Int	REMOVE ref
✓	✎ solveKey::Symbol	✎ solveKey::Symbol	label::Symbol
✓	✎ infoPerCoord::Vector{Float64}	✎ infoPerCoord::Vector{Float64}	observability
✓	✎ ismargin::Bool	✎ ismargin::Bool	marginalized
✓	✎ solvedCount::Int	✎ solvedCount::Int	solves
✓	{T} parameter	✎ variableType::String	type::String
✓		✎ _version::VersionNumber	version::VersionNumber
☐	separator::Vector{Symbol}	separator::Vector{Symbol}	separator::Vector{Symbol}
☐	val::Vector{P}		see discussion
☐		vecval::Vector{Float64}
☐		dimval::Int
☐	bw::Matrix{Float64}		see discussion
☐		vecbw::Vector{Float64}
☐		dimbw::Int
☐	covar::Vector{SMatrix{N, N, Float64}}	covar::Vector{Float64}	TODO
☐	× BayesNetOutVertIDs::Vector{Symbol}	× BayesNetOutVertIDs::Vector{Symbol}	TODO: Looks unused?
☐	× dims::Int	× dims::Int	TODO: looks deprecated
☐	× BayesNetVertID::Symbol	× BayesNetVertID::Symbol	REMOVE
☐	× dontmargin::Bool	× dontmargin::Bool	TODO: likely unused?
☐	eliminated::Bool	× eliminated::Bool	Questionable if used
☐	events::Dict{Symbol, Threads.Condition}

Keep in mind (#1177)

• that adding new fields in the future counts as feature expansion and will not break existing serialization.
• this is the chance for improving Clarity of variable names can be improved Caesar.jl#65.
  • Aim for a single noun and avoid camel case.

[Affie]

val

• Name not very descriptive, we can maybe update to reflect that this field is points on the manifolds, ie. points::Vector{P}.

Packing val::Vector{P} for serialization

Currently:

• packState converts p ∈ M to coordinates and only works for Lie groups (with a TODO referring to Serialization and Marshaling (again) #590)
• stores in fields (vecvel,dimval)

Options for a point like [[1, 2], [3 4; 5 6]]:

• Force P to be compatible with a flat column major structure SArray/MArray/ArrayPartition: [1,2,3,5,4,6]
• Nesting: [[1,2],[[3,5],[4,6]]]
  • storage? string only?
  • Possibly ambiguous?
• Complex struct coordinates plus expansion point
  • have to still store the expansion point, basis, etc.
• pack/unpack functions for points based on the variable type (such as Pose3) as
  • Vector{Float64}
  • Vector{Vector{Float64}}
  • String - base64 encoded
  • String - json value
  • String - json string

Regardless, it's maybe a good idea to add a point_encoding field to store the encoding, eg. "coords-v1", "points-v1", etc

[dehann]

Hmm, let me think on the storage of manifold points.

In the mean time, lets try clear some of the other fields that should be easier:

• dimIDs, yeah can remove. That was an idea from before JuliaManifolds existed. More likely to be replaced by infoPerCoord usage.
• eliminated is definitely used, don't remove from PackedState, and also goes along with Simultaneous solvers use same CSM / Tree? Different solves wtih different variable orderings? IncrementalInference.jl#673
• BayesNetOutVertIDs, visual inspection by search seems like unused field. My style of doing would be to remove the field and then run all tests through Caesar and then see if anything related pops up. If not, then good to remove. If really unused in all repos through Caesar, then I'm fine to remove.
• BayesNetVertID, seems used via quick search, why did you want to remove it maybe I'm missing the reasoning? (see image below). My guess is the field name is probably very old and outdated. Maybe we need to fix the name?
• _version should maybe just be version? This would be so for all types...

---

[dehann]

Dehann headscratching on serde for manifold points

Affie's whole comment: #1181 (comment)

I think anything other than String or Nested can quickly become super specific and requires a lot internal knowledge before you can unpack the data in any way. The ground rule for me remains that if you happen to come across a PackedDFG node somewhere, you can always start by loading into a dict (say in Python) that object before you know anything about it's contents json.parse(_I_found_this_file_that_looks_like.json). Now you start exploring that dict and find either a type or mime field, which you use to start Googling, hey what is Nva, CJL, or DFG, etc.

This is like the prime directive when it comes to DFG serialization. What I do know so far is that anything from the comment #1181 (comment) that violates blind json packing/unpacking is already a no go. I also quite strongly believe that any time a person somewhere in the wild starts unpacking these DFG json objects, that they want to look at the numerical estimates. So rather than hide the numbers in some obscure encoding, we should make them visible; or clearly tell them what to Google via the mime= ; encoding=... examples at the bottom of this comment.

---

This all ties in with the outstanding work of registering .nvy (.nva is taken) extension for FileDFG tars?

Regardless, it's maybe a good idea to add a point_encoding field to store the encoding, eg. "coords-v1", "points-v1", etc

is there not something like mime for manifold points, should we not register (or at least document) a bunch of mime extension options for this purpose?

---

Here is ChatGPT output on query :

`mime extension examples using ';'`

**Usage of Parameters**

Parameters can provide additional information about the content, such as character set or encoding. For example:

• text/plain; charset=UTF-8 specifies that the plain text file uses UTF-8 encoding.
• application/json; charset=UTF-8 indicates that the JSON data is also encoded in UTF-8.

Understanding these examples helps ensure proper handling of files by web servers and browsers.

---

Here is more ChatGPT output on the query ,

`existing encodings for riemannian manifold data`

Riemannian manifold data can be encoded using various methods, including persistent homology, which provides topology-inspired representations. These encodings help in analyzing the geometric and topological features of the data.[ OpenReview](https://openreview.net/forum?id=7yswRA8zzw) Overview of Riemannian Manifold Data Encodings

Riemannian manifolds are mathematical structures that allow for the study of curved spaces. Various encoding methods exist to represent data on these manifolds effectively.
Common Encoding Methods

1. Persistent Homology (PH)

   Description: PH is a topological data analysis method that captures the shape of data across multiple scales.
   Application: It generates topology-inspired representations, useful for understanding the intrinsic properties of data.
   Sensitivity: PH is sensitive to perturbations, making it effective for detecting features in data.

2. Pull-back Geometry

   Description: This novel methodology focuses on understanding the intrinsic properties of persistent homology without relying on external models.
   Advantage: It provides deeper insights into the structure of data represented on Riemannian manifolds.

Other Notable Methods
3. Geodesic Distance Encoding

Description: This method uses the shortest paths between points on the manifold to encode relationships.
Use Case: It is particularly useful in applications where the geometry of the data is crucial.

4. Heat Kernel Signature

   Description: This approach encodes the manifold's geometry by analyzing the heat diffusion process.
   Benefit: It captures both local and global geometric features, making it robust for various applications.

Conclusion

These encoding methods provide diverse ways to represent and analyze data on Riemannian manifolds, each with unique strengths and applications. Persistent homology and pull-back geometry are particularly noteworthy for their focus on intrinsic data properties.

---

Using prescribed tangent basis vectors (aka encoding coordinates with known bases)

A tangent vector can be expressed in terms of local coordinates as v=vi∂xi∂​ where vi are the components in the coordinate basis.

---

Mental steps to formalize

My sense is also that we can have more than one encoding, e.g. application/json; ABC which would be text, but say we are storing HomotopyBeliefs with 1000's of data points for high resolution belief estimation, then maybe we want to go to something like application/bson; ABC for binary representation on the bulk of data content.

So my first reaction is to maybe go for something like: mime: "application/json; [SOMETHING STANDARD]", but I don't quite know what yet... Just listing ideas:

• mime: iapplication/json; nva.tangent=se(3)
• mime: application/bson; group=nva.SGal(3)
• ...?

The other idea that comes up is to maybe, we should replace .type with .mime and rather do something like:

{
  "label": "continuation_01",
  "eliminated": true,
  # ...
  "mime": "application/json; type=DistributedFactorGraphs.PackedState, encoding=Caesar.Pose3.SGal(3)"
}

Here, I'm trying to keep the idea of different encodings for the same thing alive. So storing the raw application/json; encoding=Caesar.Pose3.SGal(3) might be really heavy but still persists. But, we can then add more encodings like application/json; encoding=RoMETypes.Pose3.sgal(3).b64 which stores only coordinates for the standard CJL bases in base64 binary format -- assuming NavAbility/Caesar.jl advertises a basis vector for sgal(3) tangents.

Or, the lightest of all would be mime: application/bson+gzip; type=DistributedFactorGraphs.PackedState, encoding=RoMETypes.RotVelPos3.sgal(3).b64, which means the whole PackedState blob exists as tightly wrapped binary rather than raw text json (exploiting the rule here that bson also requires json compliance, but stores in binary rather than text). To be clear, any json.write(jobj) can always be converted to bson using bson.write(json.parse(jstr)) and visa versa.

The problem with this is that if we start using other mimes, that needs to be known outside of the data blob. So in the real world, these payloads would only work if say the DB is known to only store application/bson+gzip, because there would be no way to know that apriori. So any data out in the wild sort of has to be application/json, small maybe application/bson but that already hurts adoption. Wild data would never be application/bson+gzip because no one would ever guess how to unpack it -- so these are for internal benefit when large data costs money, maybe things like archiving older data etc.

Additional note on why b64, because we need to stick to the rules that any DFG+Packable object is always jsonable. I.e. VariableCompute is not Packable, but VariableDFG is.

---

Recap

We can change .type to .mime and add to application/json the option ; type=DistributedFactorGraphs.PackedState as well as , encoding=CJL.Pose3.SGal(3) and the in the Caesar package advertises what that means, i.e. we use Nesting: [[1,2],[[3,5],[4,6]]] to store the whole SGal(3) affine matrix (aka group element).

We can then also advertise more efficient storage such as encoding=RoMETypes.Pose3.sgal(3) which have a documented basis with which to reconstruct the manifold point -- this asks of an uninitiated newcomer who is messing with raw data to do a bit more Googling.

[Affie]

#1181 (comment)

Sorry, I’m not fully following what you’re suggesting here.
We already have a pattern for Observation that works well, so I’d prefer not to introduce additional complexity with MIME-type variations and make things more involved than the Observation case. I’d suggest landing somewhere between the current flattened coordinate approach and a pack approach (similar to Observation).

An idea for large binary points — we already have Blobs, and if we decide to follow the existing pack/unpack pattern, we can simply store a reference to a BlobEntry (already includes the mimetype and metadata for the Blob). That way you can get your application/bson.

[Affie]

DF: sure blob approach works for large case, that also fits in with the "parch" verb usage.

I'm not sure how we'd use dedicated pack and unpack without describing the encoding somewhere? My first choice would be to avoid pack and unpack functions for manifold points, but that seems unlikely. This feels similar to the DistributionsJSON discussion, that said I like to avoid having a hard schema definition like Distributions if possible. My concern with coordinate approach is when loading to saving, that the performance of parsing and transforming all the points becomes heavy.

[Affie]

My concern with coordinate approach is when loading to saving, that the performance of parsing and transforming all the points becomes heavy.

Agree, we keep points ∈ M then.

I'm not sure how we'd use dedicated pack and unpack without describing the encoding somewhere?

We already have the <:AbstractStateType{N} (Pose2, Point2, etc.) to dispatch on. I do not think we should support more than one representation per AbstractStateType{N}.

My first choice would be to avoid pack and unpack functions for manifold points, but that seems unlikely. This feels similar to the DistributionsJSON discussion, that said I like to avoid having a hard schema definition like Distributions if possible.

I can't see a way to not have "a hard schema definition", or do you mean an extra type? I also don't see a way of not having pack and unpack functions; however, we can provide defaults for most types (eg. Matrix, ArrayPartition, Vector, etc.).

[Affie]

BayesNetVertID

BayesNetVertID, seems used via quick search, why did you want to remove it maybe I'm missing the reasoning? (see image below). My guess is the field name is probably very old and outdated. Maybe we need to fix the name?

Because it is dead code in in IIF, RoME, and Caesar

BayesNetOutVertIDs

Also dead code in IIF, RoME, and Caesar and is perhaps replaced by .separator

eliminated

eliminated is definitely used, don't remove from PackedState...

It will take a deeper dive, but it also looked like dead code.

---

DF: ah okay, I missed the dead code part. Happy to remove if dead code. I'd have to take a deeper dive to confirm, but seems unnecessary.

[Affie]

I'll confirm if these are indeed dead

[Affie]

✅ infoPerCoord -> observability

[Affie]

solveInProgress

solveInProgress::Int -> solving::Int | Number of solvers currently working on this variable (reference count). Assuming this because of the Int type

solvedCount::Int

solvedCount::Int -> solves::Int /nsolves| Total number of times this state has been updated by solvers.

---

DF:

• solves::Int is good.
• I cannot remember how solveInProgress was used, but the ::Int approach seems okay. The part where I'm stuck is massive API traffic to constantly update every variable during g a solve, so I'm questioning why have this field...

---

JT: Ok, I'll remove solveInProgress from PackedState and we can sort out what/why in DFG v1.x. (solveInProgress is also mostly dead code)

[Affie]

✅ solvedCount -> solves

[Affie]

ismargin

• maybe marginalized

dontmargin

TODO likely unused.

---

I'm fine with marginalized.

[Affie]

✅ ismargin -> marginalized

[Affie]

covar

This was prepared for parametric solve, but it is not populated yet. Maybe move to v1.1?

---

DF: just checking, if the parametric covariance value moves, that would be a breaking change - so we may have to sort this out for v1.0.

[Affie]

Good catch! We have to sort covar out as well and it fits in with the bw/val discussion.

[Affie]

bw

• Name not very descriptive, maybe more explicit as bandwidth?
  Reasons:
  • Clarity: bandwidth is immediately understandable; bw requires domain knowledge
  • Consistency: Other fields are spelled out
  • Self-documenting: The full word explains what it represents, external consumers shouldn't need to guess abbreviations.

---

DF: we should probably first resolve HomotopyBelief, which is sort out where convince/bandwidth info is stored.

[dehann]

That leaves the harder ones, val/bw/covar

we should solve these in light of future HomotopyBelief -- root of that belief tree has a mean and covariance, the leaves of that belief tree have kernels with bandwidths (or smallest singular values, depends on how we implement). I think this may result in a wholesale replacement of val/bw/covar.