Addition of metrics to track when self-healing started/ended by ShubhamRwt · Pull Request #2268 · linkedin/cruise-control

ShubhamRwt · 2025-04-22T12:29:36Z

Summary

Why: We earlier had the metrics in self healing for knowing the timestamp at which the self-healing started but it was not classified based on the anomalies. We also didn't had any metrics that can tell us when self healing was ended
What: This PR adds new metrics regarding self-healing which can show the no. of self healing started per anomaly as well as the timestamp when healing started and also we have metrics which can now tell us the self healing ended timestamp per anomaly

New metrics:

kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_broker_failure_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_disk_failure_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_goal_violation_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_maintenance_event_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_metric_anomaly_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_ended_for_topic_anomaly_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_broker_failure_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_disk_failure_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_goal_violation_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_maintenance_event_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_metric_anomaly_count
kafka_cruisecontrol_anomalydetector_num_of_self_healing_started_for_topic_anomaly_count

Categorization

new feature

This PR resolves #2266.

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ShubhamRwt · 2025-04-22T12:52:49Z

CC @ppatierno @tomncooper

tomncooper

A couple of nits.

Can you give an example output of these metrics? I'm not that familiar with dropwizard metrics. Does the dropwizard counter emit a timestamp when something is added to the internal map? I was curious how this gives us the time and type of anomaly healing being triggered.

Also, can you fix the typos in the PR description?

tomncooper · 2025-04-22T13:44:23Z

...se-control/src/main/java/com/linkedin/kafka/cruisecontrol/detector/AnomalyDetectorState.java

            MetricRegistry.name(ANOMALY_DETECTOR_SENSOR, String.format("%s-detect-to-fix-complete-timer", anomalyType.toString().toLowerCase())));
        _anomalyDetectToFixCompleteTimer.put(anomalyType, timer);
+
+        Counter counter = dropwizardMetricRegistry.counter(


If these counter and counter1 variables are not actually used anywhere other than the map.put call, can you just in-line them?

Same with timer variable above?

tomncooper · 2025-04-22T13:49:09Z

...se-control/src/main/java/com/linkedin/kafka/cruisecontrol/detector/AnomalyDetectorState.java

    _ongoingAnomalyDurationSumForAverageMs = 0;
    _numSelfHealingStarted = new AtomicLong(0L);
    _numSelfHealingFailedToStart = new AtomicLong(0L);
+    _numSelfHealingStartedPerAnomaly = new HashMap<>();


Do you need to do anything in the refreshMetrics method to reset these maps? There seems to be a numSelfHealingStarted() call in there as part of creating a new AnomalyMetrics instance.

I don't know if that is relevant but you should check if these maps need to be reset.

tomncooper · 2025-04-22T15:46:07Z

Looks like you have CI failures in the AnomalyDetectorManagerTest, did they pass locally for you?

com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest > testFixDiskFailure FAILED
    java.lang.AssertionError: On mock #1 (zero indexed): 
      Expectation failure on verify:
        ScheduledExecutorService.schedule(isA(java.lang.Runnable), 0 (long), MILLISECONDS): expected: 1, actual: 0
        at org.easymock.EasyMock.getAssertionError(EasyMock.java:2230)
        at org.easymock.EasyMock.verify(EasyMock.java:2058)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixAnomaly(AnomalyDetectorManagerTest.java:507)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixDiskFailure(AnomalyDetectorManagerTest.java:238)

com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest > testShutdown PASSED

com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest > testFixSlowBroker FAILED
    java.lang.AssertionError: On mock #1 (zero indexed): 
      Expectation failure on verify:
        ScheduledExecutorService.schedule(isA(java.lang.Runnable), 0 (long), MILLISECONDS): expected: 1, actual: 0
        at org.easymock.EasyMock.getAssertionError(EasyMock.java:2230)
        at org.easymock.EasyMock.verify(EasyMock.java:2058)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixAnomaly(AnomalyDetectorManagerTest.java:507)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixSlowBroker(AnomalyDetectorManagerTest.java:244)

com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest > testFixTopicAnomaly FAILED
    java.lang.AssertionError: On mock #1 (zero indexed): 
      Expectation failure on verify:
        ScheduledExecutorService.schedule(isA(java.lang.Runnable), 0 (long), MILLISECONDS): expected: 1, actual: 0
        at org.easymock.EasyMock.getAssertionError(EasyMock.java:2230)
        at org.easymock.EasyMock.verify(EasyMock.java:2058)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixAnomaly(AnomalyDetectorManagerTest.java:507)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixTopicAnomaly(AnomalyDetectorManagerTest.java:250)

com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest > testFixGoalViolation FAILED
    java.lang.AssertionError: On mock #1 (zero indexed): 
      Expectation failure on verify:
        ScheduledExecutorService.schedule(isA(java.lang.Runnable), 0 (long), MILLISECONDS): expected: 1, actual: 0
        at org.easymock.EasyMock.getAssertionError(EasyMock.java:2230)
        at org.easymock.EasyMock.verify(EasyMock.java:2058)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixAnomaly(AnomalyDetectorManagerTest.java:507)
        at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManagerTest.testFixGoalViolation(AnomalyDetectorManagerTest.java:232)

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ppatierno · 2025-04-30T06:32:49Z

I had a look but it looks to me we are adding kind of "historical" anomaly metrics, so for example how many anomalies (per type) started and ended. But what about what's going on in a specific time? How many anomaly fixes are actually running and on which detected anomalies?

ShubhamRwt · 2025-05-05T13:35:09Z

To know if self healing still running, we can have the sum(self healing started) minus sum(self healing ended) that can tell if the self healing is still in progress for a particular anomaly.

ShubhamRwt · 2025-05-12T07:42:42Z

@CCisGG Hi, If you get some time, can you have a look at this. It would be great to know your views on this. Thankyou

CCisGG · 2025-05-12T16:44:51Z

I'm pretty out of bandwidth recently. @mhratson @allenxwang could you please help to take a look?

ShubhamRwt added 2 commits April 22, 2025 17:58

Added new metrics

14a9a6e

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Added checkstyle fixes

db2909a

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

tomncooper reviewed Apr 22, 2025

View reviewed changes

Added suggestions by Tom and fixing the failed test

d1a2a26

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ShubhamRwt force-pushed the newMetrics branch from d235a58 to d1a2a26 Compare April 23, 2025 12:34

ShubhamRwt mentioned this pull request May 5, 2025

Proposal for adding Self healing feature in the operator strimzi/proposals#145

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of metrics to track when self-healing started/ended#2268

Addition of metrics to track when self-healing started/ended#2268
ShubhamRwt wants to merge 3 commits intolinkedin:mainfrom
ShubhamRwt:newMetrics

ShubhamRwt commented Apr 22, 2025 •

edited

Loading

Uh oh!

ShubhamRwt commented Apr 22, 2025

Uh oh!

tomncooper left a comment

Uh oh!

tomncooper Apr 22, 2025

Uh oh!

tomncooper Apr 22, 2025

Uh oh!

tomncooper commented Apr 22, 2025

Uh oh!

ppatierno commented Apr 30, 2025

Uh oh!

ShubhamRwt commented May 5, 2025

Uh oh!

ShubhamRwt commented May 12, 2025 •

edited

Loading

Uh oh!

CCisGG commented May 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ShubhamRwt commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Categorization

Uh oh!

ShubhamRwt commented Apr 22, 2025

Uh oh!

tomncooper left a comment

Choose a reason for hiding this comment

Uh oh!

tomncooper Apr 22, 2025

Choose a reason for hiding this comment

Uh oh!

tomncooper Apr 22, 2025

Choose a reason for hiding this comment

Uh oh!

tomncooper commented Apr 22, 2025

Uh oh!

ppatierno commented Apr 30, 2025

Uh oh!

ShubhamRwt commented May 5, 2025

Uh oh!

ShubhamRwt commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CCisGG commented May 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ShubhamRwt commented Apr 22, 2025 •

edited

Loading

ShubhamRwt commented May 12, 2025 •

edited

Loading