[Pitch] Compress the bundled binary data

[jspaezp]

Hello there!

I was using the crate as a dependency and noticed the bundled data is kind of big ... so I wanted to suggest gzipping it in the included bytes and decompressing that on-read.

in my setup it drops the bundled size from 25-ish MB to 8.

This is my quick and dirty implementation if you want to go over it and suggest things.
jspaezp#1

(also LMK if you have an opinion on the bundled data and version-control->binary lifecycle)

[mobiusklein]

I do something similar in my recent integration of mzcv into mzdata:

https://github.com/mobiusklein/mzdata/blob/4973ac3484ff746242dca215cd133e6ff31bcfeb/crates/mzdata-param/src/cv.rs#L604-L616

We could probably make them even smaller with better compression (zstd level 22), but it's always a tradeoff. Measured on my single bincode file for PSI-MS, it's a savings of 10KB out of 70KB vs. gzip, but gzip is "good enough".

[douweschulte]

I am happy to have the data compressed. It is indeed a bigish part of most of my compiled binaries as well, but most of them are gui and so was not really an issue for me yet. I would be most happy with a solution that does not add too much to the dependency cost.

I did read an interesting blog post recently on embedding this kind of data differently which should both compress and remove start up latency, but I have not tried this before. http://manishearth.github.io/blog/2022/08/03/zero-copy-3-so-zero-its-dot-dot-dot-negative/

For versioning, the current system is that the version information is backed in the data itself, which I like. For keeping it up to date it is just me running 'cargo mzcore-update' once in a while, and always just before any release. Release wise this might be easy to automate, but releases have been sporadic so it is not an issue at all to do it manually (which also makes it easy to detect issues).

I am in Lyon right now for IMSC so I will not be able to do a lot of code stuff this week but I am happy to keep the conversation going and even look a bit at a PR if you feel like it. Otherwise I can try and look at some options when I am back.

[jspaezp]

I did read an interesting blog post recently on embedding this kind of data differently which should both compress and remove start up latency, but I have not tried this before. http://manishearth.github.io/blog/2022/08/03/zero-copy-3-so-zero-its-dot-dot-dot-negative/

this is really not a bad idea.

For the time being I found that the compiler just leave out gnome (which is the only one large enough to annoy me) if I don't initialize it for the ontologies I pass to my proforma/mzspeclib parsing. (maybe it can be a bit better documented but not the end of the world). thanks a lot for the responsiveness!

[douweschulte]

That does sounds like a useful trick. It is always nice to rely on the compiler to do the heavy lifting. If you think it is useful I think we can also add a feature to turn off embedding GNOme in the binary this would still allow the use of GNOme via local files but should leave the binary a lot smaller. I have not yet tried the databake approach but I think I want to try it, that would also be a more readable file and not a binary blob in git so should solve your itch against binary files. On that note, I did bundle the ontologies for a long time in plain text and then built them during a build.rs step. But this took a while every time you compiled the code when the cache is gone, and at some point the GNOme data just was too big to host on github without large file storage. That all together and combined with the fact that it would only need updating once in a while I thought that binary blobs would be more effective.

[jspaezp]

(not related to this thread but wanted to raise visibility to it)
#100

LMK if you would like the PR ... noticed there is not contribution guidelines beyond the code of conduct, so let me know if contributions are accepted/welcome and how would you like that to happen.

[mobiusklein]

It's hard to use an idea like databake without effectively pushing nested Cows all the way through the type hierarchy when the same type may be static cached data OR made at runtime. The other thing is that if you use databake, then you only receive space savings iff the native type is more compact than your encoded text. For number-heavy things, that's almost certainly a yes. For strings, that's harder to prove.

It's probably better to decouple more steps in the CVSource lifecycle into make it easier to intervene and force I/O through a compression codec. The current state of bincode's maintenance is also potentially an issue, but that's a more drastic change the type hierarchy.

[douweschulte]

@jspaezp I saw the PR but took a bit longer to look into it. Contributions are definitively welcome. This is supposed to be a community project and quite some people have helped out with smaller sections but I am still doing most of the work. Please let me know if anything that I am doing is in the way of you using the library, it has been this way because I use mzcore heavily in multiple projects of mine and this is the easiest way for me to work. We have regular meetings with the rusteomics group discussing previous work, code style, solutions to shared problems and the like, the next one is wednessday 16 september at 17:00 european time. If you want to join you are welcome and I can send you a link in your mailbox, but it seems like you are also part of the group in the EuBIC slack. Feel free to discuss anything there as well I tend to keep up with the messages there. Otherwise I am happy to discuss/help/guide you in any way for this project so feel free to contact me via email or slack or even set up an online meeting to have an easy discussion. Your comment makes it clear that I should put some comment in the readme that contributions are welcome so I will get to that soon.

@mobiusklein Thanks for looking into databake, these restrictions indeed make sense, it might be better to use normal compression then. Lets discuss bincode at the next rusteomics meeting I have indeed seen the 3.0.0 'version', but figured that previous versions would keep working. I would be fine with changing to a different serialization format, it would be a bit of work but nothing out of the ordinary.

[douweschulte]

I have updated the code to compress (gz) the binary data in the final artifact. It seems to work quite well, although GNOme specifically does not compress very well. For this I added one more function to mzcv so in theory it would only need a minor version bump on all crates, but let me know if you would be interested in a new version release.

CV	pre	post	decrease (pre/post)	unit
IMGT	1536	161	9.5	KiB
GNOme	25.2	8.01	3.1	MiB
PSIMOD	673	157	4.3	KiB
Unimod	325	82.9	3.9	KiB
Resid	258	78.6	3.3	KiB
Total	28597	8682	3.3	KiB