ROS Buildfarms Buildtime Performance Analysis

[Crola1702]

Description

Buildtime between build.ros2.org and ci.ros2.org have huge differences event though we're theoretically building the same distributions. For example nightly_linux_debug took 2h 24min, while Rci_nightly_debug took 5h 8min, both building Rolling distribution

• Goal: Identify and document the root causes of buildtime differences on build.ros2.org compared to ci.ros2.org
• Objective 1: Compare hardware resources (machines), job and build configuration differences between the two farms
• Objective 2: Quantify time spent in specific build phases (Setup, Build, Test, Packages)

Requirements:

• If created, an analysis tool must be reusable for future performance monitoring (an upgrade of the current time analysis tool or a creation of a new one)

Considerations:

• Check ccache status on buildfarms
• Check parallelism in build and tests between buildfarms

Tasks:

• 0.0 Problem Statement
  • 0.1 Buildtime trend in ci.ros2.org builds
  • 0.2 Buildtime trend in build.ros2.org builds
  • 0.3 Comparison between buildtime trends
  • 0.4 Review on previous buildtime analysis tools (https://github.com/osrf/buildfarm-tools-private/pull/5)
• 1.0 Infrastructure Comparison
  • 1.1 Hardware Comparison
    • 1.1.1 Compare CPU specifications
    • 1.1.2 Compare RAM specifications
    • 1.1.3 Compare Disk I/O speeds
  • 1.2 Build Configuration
    • 1.2.1 Compare Compiler/CMake/Colcon versions
    • 1.2.2 Compare Docker runtime configurations
    • 1.2.3 Compare build and test flags
    • 1.2.4 Compare packages being built
• 2.0 Buildtime Tool Data Collection
  • 2.1 Log Analysis
    • 2.1.1 Which section is taking more time to run?
    • 2.1.2 Which package of the sections is taking more time to run?
  • 2.2 Comparisons
    • 2.2.1 Compare compilation time in build stage per packages
    • 2.2.2 Compare test time in test stage per packages
    • 2.2.3 Find bottlenecks and differences
• 3.0 Investigation closure
  • 3.1 Draft investigation document (github investigation)
  • 3.2 Formulate recommendations
  • 3.3 Present finding to Infra and ROS PMC

[Crola1702]

Buildtime trend comparison

Generated using buildtime-trend. Check the html reports by yourself:

• buildtime_line_trend_ci.ros2.org.html
• buildtime_boxplot_trend_ci.ros2.org.html
• buildtime_line_trend_build.ros2.org.html
• buildtime_boxplot_trend_build.ros2.org.html

Plots for buildtime in 2025

ci.ros2.org

Buildtime trend:

Buildtime boxplots:

build.ros2.rog

Buildtime trend:

Buildtime boxplots:

In general, is clear that build.ros2 takes more time in than ci.ros2 in general.

RMW jobs can compare to a linux repeated job in terms of time they take to build.
Also, there is a notable bump in late April, which is related to the new release and introduction of new packages into the builds.

Release and debug build generally take the same time. But Debug jobs are less consistent in the buildtime at least in build.ros2.
In ci.ros2, all linux jobs are similar in buildtime consistency (not very consistent).

[miguelgonrod]

Hardware Comparison

• ci.ros2.org: c5.xlarge
• build.ros2.org: c5.xlarge and c5a.xlarge

References: c5.xlarge & c5a.xlarge (Clara's recommendation)

CPU specifications

c5.xlarge vs c5a.xlarge::

- vCPUs	                                      4
- Physical Processor	                      Intel Xeon Platinum 8124M
- Clock Speed (GHz)	                          3.4 GHz
- CPU Architecture	                          x86_64

+ vCPUs	                                      4
+ Physical Processor	                      AMD EPYC 7R32
+ Clock Speed (GHz)                    	      3.3 GHz
+ CPU Architecture	                          x86_64

RAM specifications

c5.xlarge vs c5a.xlarge:

- Memory (GiB)	                              8
- Memory per vCPU (GiB)	                      2

+ Memory (GiB)	                              8
+ Memory per vCPU (GiB)	                      2

Disk I/O speeds

c5.xlarge vs c5a.xlarge:

- Max Bandwidth (Mbps)	           4750
- Max Throughput (MB/s)	           593.75
- Max I/O operations/second	       20000
- Baseline Bandwidth (Mbps)	       1150
- Baseline Throughput (MB/s)	   143.75
- Baseline I/O operations/second   6000

+ Max Bandwidth (Mbps)	           3170
+ Max Throughput (MB/s)	           396.25
+ Max I/O operations/second	       13300
+ Baseline Bandwidth (Mbps)	       400
+ Baseline Throughput (MB/s)	   50
+ Baseline I/O operations/second   1600

Conclusions

The main difference between both machines is the processor as one is intel and the other AMD, also the disks differ just by a little bit more speed in intel machines. Regarding the problem between ci.ros2 and build.ros2 this is not significant and not the root problem, this differences just will make as most some minutes more in time

[Crola1702]

Build Configuration Comparison

Comparison between Rci__nightly-debug_ubuntu_noble_amd64#597 (323 minutes) and nightly_linux_debug#3658 (149 minutes)

Compiler versions

build.ros2 vs ci.ros2

- Setting up cpp (4:13.2.0-7ubuntu1) ...
- Setting up cmake (3.28.3-1build7) ...
- Setting up gcc-x86-64-linux-gnu (4:13.2.0-7ubuntu1) ...
- Setting up gcc (4:13.2.0-7ubuntu1) ...
- Setting up python3-colcon-cmake (0.2.29-100) ...

+ Setting up cpp (4:13.2.0-7ubuntu1) ...
+ Setting up cmake (3.28.3-1build7) ...
+ Setting up gcc-x86-64-linux-gnu (4:13.2.0-7ubuntu1) ...
 +Setting up gcc (4:13.2.0-7ubuntu1) ...
+ Setting up python3-colcon-cmake (0.2.29-100) ...

The versions of compiler packages are the same for Ubuntu Noble containers.

Docker Version

Full docker version output in build.ros2 agents:

+ docker version
Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.23.1
 Git commit:        28.2.2-0ubuntu1~24.04.1
 Built:             Wed Sep 10 14:16:39 2025
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.23.1
  Git commit:       28.2.2-0ubuntu1~24.04.1
  Built:            Wed Sep 10 14:16:39 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.28
  GitCommit:        
 runc:
  Version:          1.3.3-0ubuntu1~24.04.3
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Difference between docker versions and docker info:

Rci__nightly_debug_ubuntu_noble (build.ros2) vs nightly_linux_debug (ci.ros2)

4,8c4,8
<  Version:           28.2.2
<  API version:       1.50
<  Go version:        go1.23.1
<  Git commit:        28.2.2-0ubuntu1~24.04.1
<  Built:             Wed Sep 10 14:16:39 2025
---
>  Version:           26.1.3
>  API version:       1.45
>  Go version:        go1.22.2
>  Git commit:        26.1.3-0ubuntu1~24.04.1
>  Built:             Mon Oct 14 14:29:26 2024
14,18c14,18
<   Version:          28.2.2
<   API version:      1.50 (minimum version 1.24)
<   Go version:       go1.23.1
<   Git commit:       28.2.2-0ubuntu1~24.04.1
<   Built:            Wed Sep 10 14:16:39 2025
---
>   Version:          26.1.3
>   API version:      1.45 (minimum version 1.24)
>   Go version:       go1.22.2
>   Git commit:       26.1.3-0ubuntu1~24.04.1
>   Built:            Mon Oct 14 14:29:26 2024
22c22
<   Version:          1.7.28
---
>   Version:          1.7.24
36c36
<  Version:    28.2.2
---
>  Version:    26.1.3
45,46c45,46
<  Images: 1264
<  Server Version: 28.2.2
---
>  Images: 136
>  Server Version: 26.1.3
60,62d59
<  CDI spec directories:
<   /etc/cdi
<   /var/run/cdi
75,76c72,73
<  Kernel Version: 6.14.0-1017-aws
<  Operating System: Ubuntu 24.04.3 LTS
---
>  Kernel Version: 6.8.0-1028-aws
>  Operating System: Ubuntu 24.04.2 LTS
80,82c77,79
<  Total Memory: 7.637GiB
<  Name: ip-10-0-1-215
<  ID: 2e1feb78-b63a-4410-89fe-cfbc636d250c
---
>  Total Memory: 7.598GiB
>  Name: ip-10-0-3-28
>  ID: e88aeb48-108c-43c9-b308-56989b49e2bf
87d83
<   ::1/128
89,90d84
<  Registry Mirrors:
<   http://repo.ros2.org:5000/

In general, build.ros2 docker software versions are higher than ci.ros2 (docker version, kernel version)

Build and test flags

Build command in ci.ros2:

/usr/bin/colcon build --base-paths "src" --build-base "build" --install-base "install" --event-handlers console_cohesion+ console_package_list+ --cmake-args -DBUILD_TESTING=ON --no-warn-unused-cli -DCMAKE_BUILD_TYPE=Debug -DINSTALL_EXAMPLES=OFF -DSECURITY=ON -DAPPEND_PROJECT_NAME_TO_INCLUDEDIR=ON

Total packages built:

Summary: 361 packages finished [1h 34min 1s]

Build command in build.ros2:

# BEGIN SECTION: Run Dockerfile - build and install
# BEGIN SUBSECTION: build workspace in isolation and install

Invoking 'PYTHONIOENCODING=utf_8 PYTHONUNBUFFERED=1 colcon build --build-base build_isolated --install-base install_isolated --test-result-base test_results --event-handlers console_cohesion+ --cmake-args -DBUILD_TESTING=0 -DCMAKE_BUILD_TYPE=Debug -DSKIP_MULTI_RMW_TESTS=1 --no-warn-unused-cli' in '/tmp/ws'

... and then:

# BEGIN SECTION: Run Dockerfile - build and test
# BEGIN SUBSECTION: build workspace in isolation
Invoking 'PYTHONIOENCODING=utf_8 PYTHONUNBUFFERED=1 colcon build --build-base build_isolated --install-base install_isolated --test-result-base test_results --event-handlers console_cohesion+ --cmake-clean-cache --cmake-args -DBUILD_TESTING=1 -DCMAKE_BUILD_TYPE=Debug -DSKIP_MULTI_RMW_TESTS=1 --no-warn-unused-cli --test-result-base /tmp/ws/test_results' in '/tmp/ws'

... and then:

# BEGIN SUBSECTION: build tests
Invoking 'PYTHONIOENCODING=utf_8 PYTHONUNBUFFERED=1 colcon build --build-base build_isolated --install-base install_isolated --test-result-base test_results --event-handlers console_cohesion+ --cmake-args -DBUILD_TESTING=1 -DCMAKE_BUILD_TYPE=Debug -DSKIP_MULTI_RMW_TESTS=1 --no-warn-unused-cli --test-result-base /tmp/ws/test_results --cmake-target-skip-unavailable --cmake-target tests' in '/tmp/ws'

Total packages built (detail of packages built in the next bullet point):

Summary: 361 packages finished [1h 2min 45s]
Summary: 361 packages finished [1h 49min 36s]
Summary: 361 packages finished [1min 38s]

Test command in ci.ros2:

# BEGIN SUBSECTION: test
/usr/bin/colcon test --base-paths "src" --build-base "build" --install-base "install" --event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"

Total packages tested:

Summary: 361 packages finished [46min 10s]

Test command in build.ros2:

# BEGIN SUBSECTION: run tests
Invoking 'PYTHONIOENCODING=utf_8 PYTHONUNBUFFERED=1 colcon test --build-base build_isolated --install-base install_isolated --test-result-base test_results --event-handlers console_direct+ --executor sequential --test-result-base /tmp/ws/test_results --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail"' in '/tmp/ws'

Total packages tested:

Summary: 361 packages finished [2h 12min 25s]

This looks like a big gap in the build stages. Both buildfarms build 361 packages (probably the same packages, but I haven't verified). However, build.ros2 builds the 361 packages twice! One in build and install stage and the other in build and test stage (the third build, build tests doesn't take much time). Is also worth noting that test stage takes almost 3 times more.

From the comparison between these two builds, it's clear that the 174 minutes difference is almost exactly because of the 77 build time and 86 testing time difference.

[Crola1702]

Build Configuration Comparison Follow Up

We performed two experiments to determine if the build structure on build.ros2.org is causing the performance gap.

Note that these tests are done in build.test.ros2.org, which uses a m5.xlarge machine, which is a less powerful machine.

1. Removing --cmake-clean-cache flag

The idea was that removing the cache between "Build & Install" and "Build & Test" stages, CMake might reuse artifacts reducing the time of the second stage.

This was done in Rci__nightly-release_ubuntu_noble_amd64 #260:

• Configuration: BUILD_TESTING=0 (Stage 1) and BUILD_TESTING=1 (Stage 2), with no --cmake-clean-cache flag.
• Results:
  • Stage 1 (Build & Install): 1h 42min
  • Stage 2 (Build & Test): 2h 22min

The build time for the second satage did not decreate. It seems that BUILD_TESTING from 0 to 1 forces the build tool to invalidate the previous state or rebuild the majority of the graph, so it doesn't seems like a good strategy.

2. Single-stage build (skipping "Build & Install")

Since the first stage artifacts are not being reused (and are wiped or invalidated), the first stage seems redundant. We tested a single-pass build similar to the ci.ros2 workflow:

This was done in Rci__nightly-release_ubuntu_noble_amd64 #267:

• Configuration: Single colcon build command with BUILD_TESTING=1 (Skipped Stage 1 entirely).
• Results:
  • Single Stage Build: 2h 39min 54s
  • Build Tests target: 1min 45s

Conclusions

Redundant Workflow: It seems that the current 3-stage workflow on build.ros2 is inefficient. The first "Build & Install" stage provides no benefit to the subsequent test stage.

Another set of comparisons (only release build types):

Buildfarm	Reference Build	Configuration	Effective Build time	Machine Type	Notes
ci.ros2	nightly_linux_release#3762	Single Stage	1h 55m	c5.xlarge	NA
build.ros2	Rci_release#598	Multi Stage	3h 5m	c5a.xlarge	1h 15m + 1h 49m + 1m
build.test.ros2	Rci_release#273	Single Stage	2h 20m	c5a.xlarge	NA

Archived experiment that wen't wrong and is misleading

Buildfarm	Build Type	Configuration	Effective Build Time	Notes
ci.ros2	Debug	Single Stage	~1h 34m	Fast reference.
build.ros2 (Original)	Debug	Multi-Stage	~2h 51m	~1h 02m (wasted) + ~1h 49m (rebuild).
build.test.ros2 (Exp #267)	Release*	Single Stage	~2h 40m	No wasted first stage (m5.xlarge machine)

Details