Add option for acknowledged Sourcing

[tehsphinx]

Proposed change

When setting up a source on a stream, it would be great to have an option to enable acknowledgement. This setting should be applied on the consumer that is used to get the data from the source stream.

Use case

Case "WorkQueue Policy"

Requirements

• Messages are meant to be delivered to a leaf node cluster with unstable connection.
• Delivery guarantee is required.
• Messages should not remain in any stream after they were delivered -- including the stream source.

Setup

• Messages are sent to a stream that buffers them to be delivered to the leaf node cluster while it is disconnected. While connected the messages pass through the buffer to the leaf node cluster.
• The buffer stream has WorkQueue policy so messages get deleted as soon as delivered.
• A stream in the leaf node cluster sources from the buffer stream.

Issue

Currently we cannot guarantee message delivery this way. Sourcing from WorkQueue cannot guarantee it since the consumer used uses AckNone policy.

Note: Using Limits policy is not a good option as the message would remain in the source stream until the limit is hit. Setting a high limit breaks guarantees given to the customer (we don't store any messages after delivery) and setting a low limit (e.g. 1h) leads to message loss if the leaf node is disconnected longer than that.

Potential solution

Assuming I understand correctly how it works: if we could choose to set AckExplicitPolicy policy on the source consumer, we would be able to use this setup and still guarantee delivery. Probably best as an option in the source config of the stream to be able to choose: delivery guarantee vs performance.

Case "Interest Policy"

Note: for us not a priority right now, but would enable powerful stream fan-out with delivery guarantee.

Same requirements as above, just multiple leaf clusters that each need the message sourced from a single stream. A kind of "fan-out" between streams -- with delivery guarantee and deletion of the message in the source once all streams have successfully sourced the message.

I guess this one might be more difficult to adjust as the consumers would need to be made durable for the "interest" to exist throughout the live-time of the sourcing setup.

Contribution

No response