A light wallet client should be able to obtain the header of each CompactBlock

[daira]

What is your feature request?
As a light wallet client, I want to have the header of each CompactBlock. That means either a way to request it, or an expectation that the light wallet server should always send the header.

How would this feature help you?
It would allow the client to verify the information provided by the light wallet server, including whether all transactions have been provided and their contents.

[LarryRuane]

If I understand correctly, you're referring to the full zcash block header that corresponds to a given compact block. This is reasonable; I have three questions:

There are currently several gRPCs that return compact blocks:

• GetBlock - returns a compact block
• GetBlockNullifiers - the same except returns only tx nullifiers, which saves data transfer
• GetBlockRange - like GetBlock except returns a stream of compact blocks within a block height range
• GetBlockRangeNullifiers - like GetBlockNullifiers except returns a stream of these trimmed compact blocks

We could add the block header to the CompactBlock gRPC protobuf schema in a backward-compatible way, so that all of the above methods would return it. The disadvantage is that if the wallet client doesn't need the block header, there is some unnecessary (wasteful) network data transfer. (If my math is correct, each header is 1484 bytes.)

Or we could add a new gRPC that just returns the header. Would we actually want two versions, one that returns a single block header and another that returns a range of headers? (Similar to GetBlock and GetBlockRange)

A separate question:

Would the best format be just the usual byte serialization of the header, or have the fields broken out?

The first option would be similar to the getblock RPC with verbosity = 0, except that returns a hex string, and returns all the block's transactions (which we wouldn't want here).

The second option would be like the getblock RPC with verbosity = 1 (or 2), where the block header fields are broken out. except without the transactions.

One last question, could we return less than the full header (to save bandwidth)? For example, do we need solution (which is the largest part of the header, 1344 bytes)? Or are all the fields needed (or may be in the future)?

[daira]

The header can already be optionally sent:

message CompactBlock {
    [...]
    bytes header = 6;                // (hash, prevHash, and time) OR (full header)

I simply want, as a client, to be able to guarantee that it is sent (past a given light wallet protocol update). As I said, that can either be a change to the protocol to specify that header always contains the full header, or a way to request it.

The disadvantage is that if the wallet client doesn't need the block header, there is some unnecessary (wasteful) network data transfer. (If my math is correct, each header is 1484 bytes.)

This ticket is satisfied if there is a way to get it. I would mildly prefer that it just always be sent (at least by GetBlock and GetBlockRange) because that's simpler, and I don't think the bandwidth saving of conditionally sending it justifies the complexity. It has to be the full header including the Equihash solution, because that's what is hashed to get the prevHash of the next header.

[str4d]

I would mildly prefer that it just always be sent (at least by GetBlock and GetBlockRange) because that's simpler, and I don't think the bandwidth saving of conditionally sending it justifies the complexity.

I disagree, and think the original documented intent of these fields (that clients have been following for six years) should be preserved. I just noticed that the OR was silently removed in zcash/lightwallet-protocol#5; that removal should be reverted.

Here are the actual costs of this proposal:

• Current behaviour: header omitted, hash, prevHash, time included (current behaviour, consistent with prior docs)
  • Empty block size: 1 + 4 + 32 + 32 + 4 + 1 + 1 + 4 + 4 = 83 bytes before protobuf overhead
  • At current height of 3173875, that is 251 MiB of CompactBlock header data that clients need to download.
• Configurable behaviour: header included, hash, prevHash, time omitted (consistent with prior docs)
  • Empty block size: 1 + 4 + 1 + 1 + 1 + 1484 + 1 + 4 + 4 = 1501 bytes before protobuf overhead
  • This incurs an 18x bandwidth increase, but controllable by each client.
  • At current height of 3173875, that is 4.4 GiB of CompactBlock header data that clients can optionally download.
• Semantic break: All fields always included (violates prior docs)
  • Empty block size: 1 + 4 + 32 + 32 + 4 + 1484 + 1 + 4 + 4 = 1566 bytes before protobuf overhead
  • This forces a 19x bandwidth increase on all clients.
  • At current height of 3173875, that is 4.6 GiB of CompactBlock header data that clients would be forced to download.

[nuttycom]

I agree with @str4d, this increase in overhead is entirely unreasonable to force on the ecosystem. Compact block download is already a massive bottleneck for light clients.

[LarryRuane]

@str4d and @nuttycom, please see zcash/lightwallet-protocol#15

[daira]

I simply said that I would be ok with either option. Since you two are not ok with one of the options, that leaves the other one: there must be a way to request the header. There currently isn't.