KAFKA-19747: Update ClientTelemetryReporter telemetry push error handling #20661

bbejeck · 2025-10-08T14:30:11Z

When a failure occurs with a push telemetry request, any exception is
treated as fatal, increasing the time interval to Integer.MAX_VALUE
effectively turning telemetry off. This PR updates the error handling
to check if the exception is a transient one with expected recovery and
keeps the telemetry interval value the same in those cases since a
recovery is expected.

Reviewers: Apoorv Mittal [email protected], Matthias
Sax[email protected]

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java

bbejeck · 2025-10-08T14:48:29Z

@apoorvmittal10 PTAL

apoorvmittal10

LGTM, thanks for the fix.

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java

mjsax

Overall, LGTM, I am just not sure if we make it too complex, tracing down the full chain for "causes"?

…g between retryable and fatal exceptions and add test cases

bbejeck · 2025-10-09T13:22:51Z

@mjsax I've updated the logic for checking exceptions
@apoorvmittal10 not sure if you would like to take another look

mjsax

Thanks for the update. LGTM.

bbejeck · 2025-10-09T20:27:04Z

Merged #20661 into trunk

…ling (#20661) When a failure occurs with a push telemetry request, any exception is treated as fatal, increasing the time interval to `Integer.MAX_VALUE` effectively turning telemetry off. This PR updates the error handling to check if the exception is a transient one with expected recovery and keeps the telemetry interval value the same in those cases since a recovery is expected. Reviewers: Apoorv Mittal <[email protected]>, Matthias Sax<[email protected]>

bbejeck · 2025-10-09T21:07:51Z

cherry-picked to 4.1 aceb32d

bbejeck · 2025-10-09T21:09:29Z

cherry-picked to 4.0 3243300

chia7712 · 2025-10-15T18:31:25Z

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java

        public void handleFailedPushTelemetryRequest(KafkaException maybeFatalException) {
            log.debug("The broker generated an error for the push telemetry network API request", maybeFatalException);
-            handleFailedRequest(maybeFatalException != null);
+            handleFailedRequest(isRetryable(maybeFatalException));


With this change, the consumer will display the following warning, even though users may not be concerned with telemetry

[2025-10-16 02:27:16,077] WARN Received unrecoverable error from broker, disabling telemetry (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)

Perhaps we shouldn't print the warning if the error is an UnsupportedVersionException. What do you think?

I agree. It's even an issue before this change, and I did raise this to @apoorvmittal10 at some point in the past...

We get something like

org.apache.kafka.common.errors.UnsupportedVersionException: The node does not support GET_TELEMETRY_SUBSCRIPTIONS

This come from https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NodeApiVersions.java#L149

I always found it very annoying.

Yeah, I agree to both. For the issue pointed by @mjsax, AFAIR, that was a common issue in code and not specific to telemetry, the log comes in debug mode. The current PR issue gets highlighted as it logs in WARN and do not differentiate with UnsupportedVersionException.

Thanks for all the feedback we will file a PR to fix the "highlight"

@chia7712 have you done this already? We'd like to get the follow-up in the upcoming 4.1.1 release, if you don't have the bandwidth right now, I can do it.

@bbejeck Sorry for the delay. @DL1231 will file a PR as soon as possible.

No worries @chia7712, thanks for the update

this was already fixed by #20722 😄

…ling (apache#20661) When a failure occurs with a push telemetry request, any exception is treated as fatal, increasing the time interval to `Integer.MAX_VALUE` effectively turning telemetry off. This PR updates the error handling to check if the exception is a transient one with expected recovery and keeps the telemetry interval value the same in those cases since a recovery is expected. Reviewers: Apoorv Mittal <[email protected]>, Matthias Sax<[email protected]>

github-actions bot added the clients label Oct 8, 2025

bbejeck commented Oct 8, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java Outdated Show resolved Hide resolved

bbejeck commented Oct 8, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java Outdated Show resolved Hide resolved

bbejeck requested a review from apoorvmittal10 October 8, 2025 14:35

apoorvmittal10 approved these changes Oct 8, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java Outdated Show resolved Hide resolved

mjsax reviewed Oct 8, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryReporter.java Outdated Show resolved Hide resolved

mjsax reviewed Oct 8, 2025

View reviewed changes

bbejeck force-pushed the KAFKA-19747_improve_failed_push_handling branch from 839a65f to 026728b Compare October 8, 2025 22:19

bbejeck added 3 commits October 9, 2025 09:14

Update ClientTelemetryReporter to handle failures by differentiatin…

a9b7f24

…g between retryable and fatal exceptions and add test cases

Move private method to later in class

4e729a9

Simplified logic for retriable

900be01

bbejeck force-pushed the KAFKA-19747_improve_failed_push_handling branch from 026728b to 900be01 Compare October 9, 2025 13:18

mjsax approved these changes Oct 9, 2025

View reviewed changes

bbejeck merged commit 1e95d04 into apache:trunk Oct 9, 2025
32 of 35 checks passed

bbejeck deleted the KAFKA-19747_improve_failed_push_handling branch October 9, 2025 20:32

chia7712 reviewed Oct 15, 2025

View reviewed changes

KAFKA-19747: Update ClientTelemetryReporter telemetry push error handling #20661

KAFKA-19747: Update ClientTelemetryReporter telemetry push error handling #20661

Uh oh!

Conversation

bbejeck commented Oct 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbejeck commented Oct 8, 2025

Uh oh!

apoorvmittal10 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

bbejeck commented Oct 9, 2025

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bbejeck commented Oct 9, 2025

Uh oh!

bbejeck commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bbejeck commented Oct 9, 2025

Uh oh!

chia7712 Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

mjsax Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

apoorvmittal10 Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

bbejeck Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

bbejeck Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bbejeck commented Oct 8, 2025 •

edited by github-actions bot

Loading

bbejeck commented Oct 9, 2025 •

edited

Loading

apoorvmittal10 Oct 15, 2025 •

edited

Loading