Jenkins "Age" metric does not reset correctly on skipped/unknown test results

[miguelgonrod]

Summary

Investigation into how Jenkins calculates the "Age" value for test results shows that the current behavior does not always reflect the real state of a test. Specifically, tests marked as "skipped/unknown" cause Age to keep incrementing instead of resetting, even when the test isn't actually failing.

How Jenkins currently calculates Age

1. Jenkins parses junitResult.xml on each build, looking for tags that mark a test as failed.
2. If a test is marked as failed, its Age counter is incremented.
3. On the next build, Jenkins checks the XML again:
   • If the test fails again, Age increments further.
   • If the entire build fails/goes red (a build regression), Age is not reset, since Jenkins can't determine whether the original failure is still present or not.
4. Age only resets to 0 once the test passes on a green build.

The bug / unexpected behavior

If a test is skipped for any reason, Jenkins deciding to skip it, the test being temporarily disabled, a "skip" parameter being added, etc. during a yellow or green build, Jenkins marks that test result as "unknown" instead of pass/fail.

• An "unknown" result does not reset the Age counter.
• If the same test is skipped again later, Age keeps increasing.
• This means a test that hasn't actually failed in a long time can still show a high/inflated Age, since skips are being (incorrectly) treated the same as failures for aging purposes.

Related: JUnit History feature

For context, the "History" graph (part of the JUnit plugin, often installed by default on Jenkins) works in a similar way to Age:

• It reads the same junitResult.xml files.
• It records pass/fail status per build to build the trend graphic.

Work done so far

• Wrote a script that calculates a "real" Age value, applying corrected conditions (i.e., not counting "unknown/skipped" results as continuations of a failure streak). The purpose isn't to take this age as the real one, but instead define which rules to calculate the age we need to the buildfarmers dashboard.
• With a small additional function, the same script can also generate the History trend manually using matplotlib, independent of the plugin's own rendering.

[miguelgonrod]

Case 1

Reference build: nightly_win_rep#4174
Failing test: test_communication.test_messages_c__rmw_connextdds.gtest.missing_result
Report it appeared: https://github.com/osrf/buildfarm-tools/issues/266#issuecomment-5045600613

This job says that this test has an age of 119, but checking the work of junit this doesn't mean is a consecutive error.
Junit calculates the age using this formula: first time it appeared - actual job +1

Reference Code: https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/CaseResult.java

But, as I understood there are still some conditions that can restar the age counter. Im still checking which conditions are those

[miguelgonrod]

Case 2

Reference build: nightly_linux_repeated#4351
Failing test: test_tracetools.test_message_link_partial_sync.xunit.missing_result
Report it appeared: https://github.com/osrf/buildfarm-tools/issues/268#issuecomment-5103986483
Age claimed: 153

As we can see, the first appearance of this test according to jenkins is in 4199, and this one is 4351. 4351 - 4199 = 152 + 1 = 153.

But here is something special to add. There are some tests like this that when there is no change they don't report anything (Need more context of why), when this happens Jenkins count this as if it didn't run, this explains the test jumps in builds.
Eg: nightly_linux_repeated#4350

7: test/test_message_link_partial_sync.py .Warning: No data was collected. (no-data-collected); see https://coverage.readthedocs.io/en/7.13.5/messages.html#warning-no-data-collected
7: WARNING: Failed to generate report: No data to report.
7: 
7: Warning: Failed to generate report: No data to report.
7: 
7:                                                                          [100%]
7: 
7: - generated xml file: /home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_tracetools/test_results/test_tracetools/test_message_link_partial_sync.xunit.xml -
7: 
7: ---------- coverage: platform linux, python 3.14.4-final-0 -----------
7: 
7: ============================== 1 passed in 2.01s ===============================
7: -- run_test.py: return code 0
7: -- run_test.py: verify result file '/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_tracetools/test_results/test_tracetools/test_message_link_partial_sync.xunit.xml'
 7/42 Test  #7: test_message_link_partial_sync ..................   Passed    2.98 sec

Here we can see that it says passed but in reality it didn't generated a report

There are lots of passes in between, and builds failed, so it is right that this only calculates the time since this showed up the first time

Case 3

Reference Build: https://ci.ros2.org/job/nightly_linux-aarch64_repeated/3639/
Failing test: projectroot.test_tutorial_parameter_events__rmw_connextdds
Report it appeared: https://github.com/osrf/buildfarm-tools/issues/268#issuecomment-5091413283
Age claimed: 3

This case shows that in theory jenkins age does calculate the sequential using its own rules

Case 4

Reference Build: https://ci.ros2.org/job/nightly_linux-aarch64_release/3635/
Failing test: test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time
Report it appeared: https://github.com/osrf/buildfarm-tools/issues/268#issuecomment-5091413283
Age claimed: 6

This is another exceptional case, I have detected a peliculiar behavior from Junit, it mixes tests names even if they are different proyects
Eg: projectroot.very_important_test and test.very_important_test will mix their ages.
This make sense for some tests that surely are related, as an example rclcpp.test_time_source.gtest.missing_result happens because projectroot.test.rclcpp.test_time_source failed and it didn't returned and output so they are related, but this isn't always the case.

With that explanation, this is what is happening in this case.
This tests began to happen in https://ci.ros2.org/job/nightly_linux-aarch64_release/3630, doing the formula 3635 - 3630 = 5 + 1 = 6 which makes sense, but the history trend doesn't show that error because this test is running multiple times, in 3630 it failed 5 times with the same name (Not sure why 5 times with the same name, this should be asked to ROS PMC)

So they are being treated by jenkins as different test regressions, but when Junit does the age mix I explained before they have been failing consecutively 6 builds

[Crola1702]

A couple of question to clarify the cases:

1. When a test doesn't have a JUnit report (like case 1 and case 2), is it marked as skipped by Jenkins or by colcon? Why are these test not running (e.g., an specific instruction or mark in the code)? Can we get somehow the exact builds where a test like this was run? Why is the JUnit result not reported?
2. What's the difference between the multiple test runs? (In case 4, are there multiple runs for the same test, do they differ in anything?)

[miguelgonrod]

1. I'm not sure how to check but I would say it is by jenkins, as I checked there are two cases cases of skip, first is really marked as skip, and second one is when it is marked as passed by jenkins but there is no output so Junit interpret it as skipped, because Junit parses tests output to verify age and status, instead of relying on some native Jenkins metric. Regarding why isn't running I can't find something specific, for case 2 it does run, but I need to analyze how that specific test works because analyzing the behavior it seems that when there is no change in the output between last run and actual run it prefers not to give an output, just the warning Warning: Failed to generate report: No data to report.. We can check if a build was skipped, passed, failed etc... like this https://ci.ros2.org/job/nightly_linux-aarch64_release/3635/testReport/junit/test_security/TestSecurePublisherSubscriber/test_subscriber_terminates_in_a_finite_amount_of_time/api/json, but it returns jenkins status, so when it passed (again as case 2) but with no output it will be marked as passed instead of the false skipped, so if we want to know if really passed we would need to develop an script that verifies if that warning pattern is present
2. The multiple runs are not duplicates of the same test, they are separate CTest invocations that happen to share the same reported test name.
   The Python unittest method test_subscriber_terminates_in_a_finite_amount_of_time (class TestSecurePublisherSubscriber) is executed once for every combination of:

• message type (Empty, UnboundedSequences, ...)
• RMW implementation (rmw_connextdds, rmw_cyclonedds_cpp, rmw_fastrtps_cpp, ...)
• security scenario (non_secure_comm, secure_comm, secure_not_connecting)
• repetition index (_0, _1, _2, ...)

Each combination is a distinct CTest target with its own unique name, e.g.:

test_secure_publisher_subscriber__Empty__rmw_connextdds__secure_comm_2
test_secure_publisher_subscriber__Empty__rmw_connextdds__secure_comm_3
test_secure_publisher_subscriber__UnboundedSequences__rmw_connextdds__secure_comm_0

However, each of these CTest targets internally runs the same generated Python test file, and that file always reports results under the same classname.testname: test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time. So in the JUnit/xunit output and therefore in the Jenkins dashboard they all collapse onto what looks like one test name, even though they're really different binaries, different RMW implementations, and different security configurations being exercised.
On top of that, in the failing runs I inspected (rmw_connextdds with secure_comm_2/3 and secure_not_connecting_0/1), test_subscriber_terminates_in_a_finite_amount_of_time itself actually passed (ok). The failure came from a different method in the same generated file, test_processes_finished_gracefully (class TestSecurePublisherSubscriberAfterShutdown), which failed because rmw_connextdds couldn't create a DDS participant, causing test_subscriber to exit with code 1. Since both methods live in the same file/testReport entry, it's worth double-checking whether Jenkins is actually attributing the failure to test_subscriber_terminates_in_a_finite_amount_of_time specifically, or lumping it in with test_processes_finished_gracefully under the same suite.

This is one of the situations I meant in case 4, that this test is really an error that depends on other regression, but as this output is the same name even if they are caused by connext, zenoh, cyclone, etc... the mix of age don't work as we expected, when this happens I have to check the root regression.