File tree Expand file tree Collapse file tree 1 file changed +17
-0
lines changed 
doc/architectural_decisions Expand file tree Collapse file tree 1 file changed +17
-0
lines changed Original file line number Diff line number Diff line change @@ -34,6 +34,23 @@ documents allowed by `event-model`
3434Wherever Kafka is mentioned above, the actual implementation may be a Kafka-like (e.g. RedPanda). 
3535``` 
3636
37+ ### Alternatives considered  
38+ 
39+ Encoding bluesky documents into JSON and then wrapping them in the
40+ [ ` json_json.fbs `  flatbuffers schema] ( https://github.com/ess-dmsc/streaming-data-types/blob/58793c3dfa060f60b4a933bc085f831744e43f17/schemas/json_json.fbs ) 
41+ was considered.
42+ 
43+ We chose ` msgpack `  instead of json strings + flatbuffers because:
44+ -  It is more standard in the bluesky community (e.g. it is the default used in ` bluesky-kafka ` )
45+ -  Bluesky events will be streamed to a dedicated topic, which is unlikely to be confused with data
46+ using any other schema.
47+ 
48+ Performance/storage impacts are unlikely to be noticeable for bluesky documents, but nonetheless:
49+ -  ` msgpack ` -encoded documents are 30-40% smaller than ` json `  + flatbuffers
50+ for a typical bluesky document
51+ -  ` msgpack ` -encoding messages is ~ 5x faster than ` json `  + flatbuffers encoding
52+ for a typical bluesky document.
53+ 
3754## Justification & Consequences  
3855
3956We will stream bluesky documents to Kafka, encoded using ` msgpack-numpy ` .
    
 
   
 
     
   
   
          
     
  
    
     
 
    
      
     
 
     
    You can’t perform that action at this time.
  
 
    
  
     
    
      
        
     
 
       
      
     
   
 
    
    
  
 
  
 
     
    
0 commit comments