Deprecate and replace "durable" for consumers.

[scottf]

Proposed change

Over time, the behavior of a "durable" consumer has changed, which can cause confusion as what the current actual behavior is.

The proposal is to deprecate "durable" naming in both the server api and the clients, and replace it with flags or other appropriate configuration that clearly indicate the actual behavior.

Use case

By deprecating, we do not break any systems or api contracts and indicate that this will be removed in a major version. By adding replacements we clarify behavior.

Contribution

No response

[Jarema]

The challenge is that the Consumer without Durable automatically has inactive_threshold set to 5s.
Plus we have quite a few APIs for Consumer creation.

[scottf]

Plus we have quite a few APIs for Consumer creation.
This would be a change to the consumer configuration, not an api call change.

The challenge is that the Consumer without Durable automatically has inactive_threshold set to 5s.

We have already abused durable by allowing the user to set an inactive threshold override, making it technically not durable in the original sense. Which is why it would be useful to actually enumerate actual behavior for a consumer that sets the durable name has versus one that does not have the durable name set.

What exactly is the difference? We know it's not just about the threshold.

[Jarema]

I agree that using Name would be better, and deprecating Durable would be good, but how you want to make it without breaking changes?

If you pass a consumer config without durable (just with name), server will automatically set inactive threshold == 5s.
If we change this behavior, suddenly there will be dozens on consumers not cleaned up.
If we don't - consumer will get wiped.

[jnmoyne]

I also don't like that we now have both Durable and Name, I would personally be perfectly happy just to have Name but the problem is that there is currently no nice way to specify an 'infinite' or 'never' inactive threshold other than some very large number of nanoseconds, so we would need to add some custom JSON marshaling to allow some nicer value to specify it's a durable, especially since we can't change the current behavior where it defaults to 5s if not specified.

[scottf]

@Jarema My definition of deprecated is that it still be used and there is zero behavior change. This means it is not a breaking change, but might be removed in any major version.

There would be zero behavior change when durable name is not provided, I was never suggesting that ephemeral behavior be changed. The only thing I'm suggesting is adding granular fields specifically mapped to behavior already found in a durable.

So what are the actual differences between durable and ephemeral? I think this is the starting point and needs to be clearly identified and enumerated. I asked this question in slack and got 1 answer that was "I think this is the difference". Who knows the actual answer to this question? @neilalexander @MauriceVanVeen

My current knowledge is (please correct)

• By default a durable has an infinite inactive threshold, but this can be overridden and is why we are here in the first place.
• An ephemeral will only be in-memory, so it is essentially always mem_storage
• mem_storage forces a "durable" to memory only.

? Does an ephemeral adhere to the num_replicas or is it not replicated?

As an example of a granular field, we could have a bool field that is infinite or simply allow an inactive threshold set to be max int64 nanos (the client would provide either via api). Maybe we need a better name than infinite.