Add versioning to signatures

[williballenthin]

As I read the FB file format, I saw that there are still some areas for improvement (naturally!) yet the format doesn't seem to have a version field. Has this been considered yet? I'm afraid if it hasn't then it will be challenging to know which signatures can be used with which Warp implementation (of course today there is just one, but...).

[emesare]

Originally we had a version tag for each entry, I am deciding where to actually put the version. Storing it for each individual Function is probably where it should be added, and was where it was originally.

warp/signature.fbs

Lines 33 to 39 in bae7379

	table Function {
	// TODO: Add architecture, this is required to support multi-arch binaries...
	guid:string (required);
	symbol:SymbolBin.Symbol;
	type:TypeBin.Type;
	constraints:FunctionConstraints;
	}

If we want to version just the file then would be enough to add it here

warp/signature.fbs

Lines 41 to 44 in bae7379

	table Data {
	functions:[Function];
	types:[TypeBin.ComputedType];
	}

It should be noted that the current files are compressed and the same idea could be applied there (i.e. to use the version id to denote compressed vs uncompressed):

warp/rust/signature.rs

Lines 120 to 121 in bae7379

	// Move this to Data spec enum or something so that in the future we can do uncompressed versions.
	let mut encoder = flate2::write::GzEncoder::new(Vec::new(), Compression::default());

With that being said, the easiest thing to do would be to add a version field to the Function table. If we want to version whole files at that point I think it would also be a good idea to nest the Data table in another table that describes the transforms on the raw bytes (i.e. is it compressed, and if so, what compression algorithm).

table File {
    version:FileVersion;
    compress:CompressionType;
    data:[ubyte];
}

[emesare]

I'm afraid if it hasn't then it will be challenging to know which signatures can be used with which Warp implementation

That is an interesting view on the use of WARP, currently our goal is to align the signatures for use in all tools. But as more things come up it seems as though that is possibly a pipe dream (at-least for a subset of use cases).

Following that train of thought it might be beneficial to not only label a simple version, but provide some way to say "What did we mask", an example:

Say that a tool cannot identify effective NOP's (i.e. they cannot mask setting a register to itself) they would be unable to use signatures that did apply at least once, that mask operation, but they would totally be fine using any other signature that never made use of that operation.

If we had a bitflag field where each distinct mask operation sets a bit we could quickly filter out signatures that are unsupported by the consumer integration. I am not exactly sure how this would work when we start networking signatures, I would expect some handshake to occur that would give the supported operations to the server as a filter.

Anyways this is just a thought seperate from adding a version field.

[jcspencer]

Just to chime in here, the good thing about FlatBuffers is that you can always add new fields to the end of structures, and they'll be ignored by older parsers. You can deprecate old fields too, but you can't really remove anything - or add new fields at the "top" of structures.

Adding an version: uint16 & enum Flags: uint64 to the start of each structure could be a nice way to future-proof things so you don't have to tack them on at the end later?

[emesare]

Yea some of that exists in the type spec, at-least when it comes to the flags. I am going to add a version enum to the Function table when I am back from PTO.

I can't really think of any other place or considerations for not adding it and we need it!

Aside from that I also am trying to fix the mess that is the on-disk format.

[emesare]

One thought I just had is maybe we have a u16 integration ID and a version ID

so for binja a function would be:
Function (version 0:01)
And for let's say Ghidra it might be:
Function (version 1:01)

That way we can allow each integration to change matching behavior given a particular integration. Of course in an ideal world we would all have identical matching algorithms, but at least this would give us an escape hatch.

[emesare]

As for adding flags I think that is a separate discussion, although like you mention falls under future proofing so in that sense it's a similar thing.

So if we added:
Integration ID: u16
Version ID: u16
Flags: u32 sized enum

I think that would be very flexible.

[emesare]

Now that I think about it more we should keep integration ID 0 reserved for "unspecified" so that we can emit functions directly without talking about an integration.

[emesare]

Closed with #30