KV.Update errors with "wrong last sequence" (10071) with bucket replica=3 only under high update frequency

[michaelwitt13]

Observed behavior

We have a NATS Cluster consisting of three nodes. Under high update frequency of updating a KeyValue with KV.update with the NATS Java driver using always the right revision by the client for each KV.update method call, there are a lot of JetStreamApiExceptions emitted with the error message "wrong last sequence" (10071) by the NATS server.
The errors do not appear if the KV bucket is configured with replica=1 but it appears very often (hundreds / thousands of times) when the bucket is configured with replica=3.
Is this a known issue?
Is the KV.update designed to be stable under high load and frequency in a replica-scenario?
Is this a bug or would you advise against using replicas in a high load / frequency scenario updating KeyValues at all?

Expected behavior

If a NATS client like the Java driver calls KV.update with the correct revision, the server / driver should not reply sporadically with an error message "wrong last sequence (10071)". If KV.update is called with the right revision and the NATS cluster is up, it should always update without any error, even under a high load / high frequency scenario.

Server and client version

nats_server_version: 2.11.8-alpine
Java driver version: io.nats:jnats:2.21.4

Host environment

Kubernetes infrastructure with 3 NATS cluster nodes running on AWS

Steps to reproduce

No response

[MauriceVanVeen]

The server will respond with that error message if you'd propose two of the same kv.update calls in parallel. The first one will go through, but the second one would trigger this error message (rightfully so).

Can you confirm whether that's happening here as well?

With an R1 you would not get wrong last sequence, but only wrong last sequence: {seq} in that case. The former error without a known sequence can happen if the stream is replicated and the kv.update is proposed but hasn't been accepted by a quorum of servers yet.

[michaelwitt13]

Thanks for the quick response.

In this case, there may be parallel updates on the same bucket, but only on different KeyValues by different client threads. For the same Key the KV.updates are are always coming in sequence with the right revision. This is guaranteed and reviewed by the client code.

In my cases I always get errors "wrong last sequence" 10071 with a sequence given in the error message. In my case there is no error message without the sequence given.

My tests show clearly that with replicas=3 there are lot of errors with error message "wrong last sequence" [10071] with a sequence number given in the error message. So my tests show that under high load / update frequency the KV.update replies with those kind of error messages.

Example error stacktrace:

i.n.c.JetStreamApiException: wrong last sequence: 61406293 [10071]
	at i.n.c.api.ApiResponse.throwOnHasError(ApiResponse.java:89)
	at i.n.client.api.PublishAck.<init>(PublishAck.java:44)
	at i.n.c.impl.NatsJetStream.processPublishResponse(NatsJetStream.java:182)
	at i.n.c.impl.NatsJetStream.publishSyncInternal(NatsJetStream.java:154)
	at i.n.c.impl.NatsJetStream.publish(NatsJetStream.java:92)
	... 32 frames truncated (including 27 common frames)

[MauriceVanVeen]

Could you share a reproducible example?

[michaelwitt13]

It would be hard to do that because the complete setup which is producing the high load consists of several components:

• A Jenkins with a k6 pipeline producing load on service A, 3 parallel runs for 3 different accounts, each with 120 requests-per-seconds.
• The service A using NATS to publish messages to the first JetStream.
• The NATS Cluster which is exporting the NATS message to a 2nd JetStream.
• The service B using two durable consumers on the 2nd JetStream pull-subscribing the messages with connection X and Z.
• The bucket holding the KeyValues which must be updated (either with replica=1 or replica=3?)
• The service B consuming each message and counting each one by updating the account counters with connection Y on the bucket. Each account has its own KeyValue. All messages coming in sequence. For performance optimization not all counters are updated to KV. Most updates are done in a cache to make it faster.
  Which component(s) would be sufficient for you to reproduce? I have no separated high-load setup I could share to you.
  It is closed code of our company.

[michaelwitt13]

Will the new feature of Counters (see here: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-49.md) make a high load / high frequency scenario (with or without replicas) make faster and or reliable?

[MauriceVanVeen]

Which component(s) would be sufficient for you to reproduce?

A small shareable example that reproduces the wrong last sequence: {seq} case would help greatly.

[michaelwitt13]

Could you share a reproducible example?

I have made a smaller example which is a Gradle Java project, a docker compose file for the NATS Cluster and a CMD script providing the publishing input via nats bench commands. When zipped, it has about 80 KB. Do you have a good way sharing it to you privately?

[MauriceVanVeen]

I have made a smaller example which is a Gradle Java project, a docker compose file for the NATS Cluster and a CMD script providing the publishing input via nats bench commands. When zipped, it has about 80 KB. Do you have a good way sharing it to you privately?

Nice! If you could send it to maurice@synadia.com

[MauriceVanVeen]

I've received and reviewed the reproducible example, thank you!
Sadly it doesn't seem to reproduce for me. However, I'm currently thinking the error is genuine and a result of having Direct Get enabled on the KV.

When Direct Get is enabled any kv.get(..) request can be answered not only by the stream leader, but also by its followers. This means a follower could temporarily be just outdated enough to serve a response with an outdated sequence. When you use that as the revision in kv.update(..) you'd get the error message you're seeing. The sequence/revision would eventually converge to the sequence you're expecting and need for the KV update.

Could you run the following command on the KV to disable Direct Get, and then re-run your repro to see if it doesn't result in those wrong last sequence errors?

nats str edit -a --no-allow-direct

[michaelwitt13]

I tried and tested it with disabled Direct Get on the KV Bucket. My test is now successful.

Should I worry about the warning message when I change the options on the KV Jetstream?

Differences (-old +new):
  api.StreamConfig{
        ... // 24 identical fields
        DenyPurge:     false,
        RollupAllowed: true,
-       AllowDirect:   true,
+       AllowDirect:   false,
        MirrorDirect:  false,
        DiscardNewPer: false,
        ... // 7 identical fields
  }
WARNING: Operating on the underlying stream of a Key-Value bucket is dangerous.

Key-Value stores are an abstraction above JetStream Streams and as such require particular
configuration to be set. Interacting with KV buckets outside of the 'nats kv' subcommand can lead
unexpected outcomes, data loss and, technically, will mean your KV bucket is no longer a KV bucket.

Continuing this operation is an unsupported action.

[MauriceVanVeen]

Thanks for confirming. That means you're relying on read-after-write guarantees, which JetStream currently doesn't offer.

Direct Get allows followers to answer read requests, which means you can get a stale read like you have observed. Disabling it means only the KV/stream leader answers read requests. This fixes >99% of the cases, but is not fully watertight (an old stream leader could technically answer under niche conditions). 100% consistent support for it will be done as part of this issue: #6557

Should I worry about the warning message when I change the options on the KV Jetstream?

Generally you should not (need to) change the KV config at all. But, disabling Direct Get is currently a viable workaround for getting more consistent reads. So it's perfectly fine to change purely that setting. Once the above issue gets fixed, we'll have a better solution that doesn't involve a workaround.