Skip to content

Commit dc2c2cc

Browse files
committed
Mention we considered json_json.fbs
1 parent 1327136 commit dc2c2cc

File tree

1 file changed

+17
-0
lines changed

1 file changed

+17
-0
lines changed

doc/architectural_decisions/009-kafka-streaming.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,23 @@ documents allowed by `event-model`
3434
Wherever Kafka is mentioned above, the actual implementation may be a Kafka-like (e.g. RedPanda).
3535
```
3636

37+
### Alternatives considered
38+
39+
Encoding bluesky documents into JSON and then wrapping them in the
40+
[`json_json.fbs` flatbuffers schema](https://github.com/ess-dmsc/streaming-data-types/blob/58793c3dfa060f60b4a933bc085f831744e43f17/schemas/json_json.fbs)
41+
was considered.
42+
43+
We chose `msgpack` instead of json strings + flatbuffers because:
44+
- It is more standard in the bluesky community (e.g. it is the default used in `bluesky-kafka`)
45+
- Bluesky events will be streamed to a dedicated topic, which is unlikely to be confused with data
46+
using any other schema.
47+
48+
Performance/storage impacts are unlikely to be noticeable for bluesky documents, but nonetheless:
49+
- `msgpack`-encoded documents are 30-40% smaller than `json` + flatbuffers
50+
for a typical bluesky document
51+
- `msgpack`-encoding messages is ~5x faster than `json` + flatbuffers encoding
52+
for a typical bluesky document.
53+
3754
## Justification & Consequences
3855

3956
We will stream bluesky documents to Kafka, encoded using `msgpack-numpy`.

0 commit comments

Comments
 (0)