feat(kao): add support for bao kao testing framework

[danielRep]

Overview

This PR integrates Bao Kao (abbreviated as bkao, "to put to the test"), an end-to-end testing (and benchmarking) framework for the Bao Hypervisor. Bao Kao automates the full pipeline from source build to on-target execution: it fetches toolchains, builds guest workloads, generates Bao configuration files based on targeted setup from YAML descriptors, assembles a bootable image (with optional firmware), launches it on the target platform (emulated or physical), and captures serial output to determine pass/fail.

Two new submodules are introduced:

• tests/bkao: submodule containing the Bao Kao framework
• tests/benchs: OSYX close-source benchmarks

Platforms supported:

• Virtual:
  • qemu-aarch64-virt
  • qemu-riscv64-virt
  • fvp-a
  • fvp-r
• Physical
  • zcu104
  • s32z270
  • tc4dx
  • rh850 (partially)

Note

This PR supersedes #154. We have refactored completely the framework, removing nix support and leveraging only python to implement bao kao.

Source Tree Structure

  tests/
  ├── bkao/                   # Bao Kao framework (submodule)
  │   └── src/
  │       ├── bkao.py         # Main runner and orchestration entry point
  │       ├── platforms/      # Per-platform build and launch logic
  │       ├── firmware/       # ATF, OpenSBI, U-Boot constructors
  │       ├── guests/         # Guest workload builders (baremetal, ...)
  │       ├── hypervisor/     # Hypervisor builders including Bao and a config generator
  │       └── utils/          # Helpers
  ├── benchs/                 # Benchmarks workload repository (submodule)
  │   ├── src/benchmarks/
  │   │   ├── ctx-switch/     # Context-switch benchmark
  │   │   └── irq-lat/        # IRQ latency benchmark
  │   └── configs/            # Per-benchmark YAML platform configs
  └── tests/
      ├── configs/
      │   ├── baremetal/      # Baremetal guest configs (YAML + platform BSPs)
      │   ├── freertos/       # FreeRTOS guest configs
      │   └── linux/          # Linux guest configs
      └── src/
          ├── inc/            # Test framework headers (testf.h, asserts, commands)
          ├── boot.c          # Boot test
          ├── irq.c           # IRQ test 
          └── bao-test.mk     # Test build integration makefile

Tests Available

┌─────┬────────────┬──────────┬───────────┬──────────────────────────────────────────────┐
│ ID  │   Suite    │   Test   │   Setup   │                 Description                  │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 100 │ BOOT_CHECK │ VM_BOOT  │ baremetal │ Check that baremetal guest boots             │
│     │            │          │           │ successfully                                 │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 101 │ BOOT_CHECK │ CPU_BOOT │ baremetal │ Check that all CPUs on the baremetal guest   │
│     │            │          │           │ boot successfully                            │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 200 │ IRQ_CHECK  │ TIMER    │ baremetal │ Check that timer interrupt is triggered and  │
│     │            │          │           │ handled successfully                         │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 201 │ IRQ_CHECK  │ UART     │ baremetal │ Check that UART interrupt is triggered and   │
│     │            │          │           │ handled successfully                         │
└─────┴────────────┴──────────┴───────────┴──────────────────────────────────────────────┘

Tests are defined using the BAO_TEST(suite, test, setup, description) macro and discovered automatically at runtime by scanning tests/tests/src/*.c. Each test is tagged with a suite, a setup (which selects the VM configuration to use), and an ID used for selective execution.

IDs are computed as (file_index × 100) + test_index_within_file, where:

• file_index — 1-based position of the .c file in alphabetical order within
  tests/tests/src/
• test_index — 0-based order of BAO_TEST(...) appearances within that file

So with the current files (boot.c, irq.c):

┌────────┬────────────┬─────────┬─────┐
│  File  │ file_index │ test_nr │ ID  │
├────────┼────────────┼─────────┼─────┤
│ boot.c │ 1          │ 0       │ 100 │
├────────┼────────────┼─────────┼─────┤
│ boot.c │ 1          │ 1       │ 101 │
├────────┼────────────┼─────────┼─────┤
│ irq.c  │ 2          │ 0       │ 200 │
├────────┼────────────┼─────────┼─────┤
│ irq.c  │ 2          │ 1       │ 201 │
└────────┴────────────┴─────────┴─────┘

Guest types supported in test configs: baremetal, freertos, linux. For now, we only target a baremetal setup.

How to Run (TLDR)

Via bao Makefile targets:
Run all tests for a platform:
make tests PLATFORM=<platform>

Run all benchmarks for a platform:
make benchs PLATFORM=<platform>

Other options only via calling bkao-py directly:

Run specific tests or benchmarks by ID:
python3 tests/bkao/src/bkao.py -t 100 101 -p <platform>
python3 tests/bkao/src/bkao.py -b 100 -p <platform>

Run all tests except some:
python3 bkao.py -t -x 200,201 -p <platform>

Skip firmware and toolchain rebuild (faster iteration):
python3 bkao.py -t -p <platform> --no-firmware-build --no-toolchain-build

Use custom hypervisor sources:
python3 bkao.py -t -p <platform> --hyp-srcs /path/to/bao

Set verbosity (0=final report only, 1=failures, 2=all):
python3 bkao.py -t -p <platform> -l 2

Pass platform-specific args - only virtual platforms (e.g. GIC version):
python3 bkao.py -t -p qemu-aarch64-virt --plat-virt-args "GICV3"

Bao Kao Process Pipeline

For each test or benchmark, bkao executes the following pipeline:

1. Discover: scans tests/tests/src/*.c for BAO_TEST macros, benchs/src/benchmarks/ for benchmark dirs, and platforms/*.py for platform builders.
2. Parse args: reads CLI arguments, resolves which tests/benchmarks to run, and builds the runtime config.
3. Platform setup: instantiates the requested platform class and calls setup_platform() (emulator paths, serial config, etc.).
4. Toolchain: builds or fetches the cross-compilation toolchain; skipped with --no-toolchain-build, expecting the prefix to be in PATH.
5. launch_tests(): for each group of tests/benchmarks sharing the same setup:

• Renders config.c from the YAML VM descriptor via Jinja2 templates.
• Builds each guest workload (baremetal, FreeRTOS, Linux, or benchmark binary).
• Builds the Bao hypervisor against the generated config and guest images.
• Builds firmware (ATF/OpenSBI/U-Boot); skipped with --no-firmware-build.
• Launches the platform and captures serial output, parsing [TESTF-C] tokens for pass/fail.

6. Final cleanup: removes wrkdir/ build artifacts.

[josecm]

Question, not a change request: what do we get from the yaml + jinja over just writing config.c?

One yaml per (setup, platform) is one config.c per (setup, platform), so it's not fewer files, and nothing in the output is actually dynamic. Every value comes from the yaml, and BAO_WRKDIR_IMGS is resolved by the preprocessor, not the template. It's also missing things we'd want to test: cpu_affinity, colors, ipcs, remio_devs, mmu. And s32z270 already ships a hand-written plat.c next to its yaml.

Is the idea (a) to describe VMs separately and compose them for multi-guest setups, or (b) to be able to generate configs for another hypervisor? args[2].split("+") already builds a guest list so (a) looks intended, but the config lookup is still per-combination. If it is (a), does it also do the allocation (non-overlapping regions, cpu partitioning)? Just concatenating two VM configs doesn't really save anything.

Either way it should be in the README, because from outside it isn't obvious what it's for.

[Diogo21Costa]

This was a design choice; its main purpose is to make a test scenario a single, declarative unit: the Bao configuration together with the guest/build parameters needed to reproduce that scenario. In that sense, a YAML file describes more than the contents of config.c. It is intended to be the input consumed by the test/build tooling, while the generated config.c is one target-specific artifact of that description. This also gives us a path to emit equivalent scenario configurations for other hypervisors, rather than making the benchmark/test definitions inherently Bao-specific.

That said, the current implementation does not yet fully realize the compositional model you describe:

• args[2].split("+") was introduced with multi-guest scenarios in mind, but configuration selection is still effectively per (setup, platform) combination.
• It does not currently compose independently defined VM descriptions and perform resource allocation or validation (for example, CPU partitioning, non-overlapping memory regions, cache colors, IPCs, remote-I/O devices, or MMU settings).

So the intended answer is closer to (a) as a longer-term scenario-description direction, plus (b) as an eventual portability benefit, rather than a claim that the current YAML files already provide full VM composition or allocation. Today, they mainly centralize the scenario metadata/configuration used by the test infrastructure and generate a Bao-specific configuration from it.

[josecm]

@Diogo21Costa can you go comment by comment and say which points of the review are addressed by the latest push, and if relevant, provide futher information. For those not addressed and we which you don't plan to address, please explain we don't think they should be.

[Diogo21Costa]

@Diogo21Costa can you go comment by comment and say which points of the review are addressed by the latest push, and if relevant, provide futher information. For those not addressed and we which you don't plan to address, please explain we don't think they should be.

@josecm Matched each of the comments with the specific fixes. While validating the review changes, make tests PLATFORM=qemu-riscv64-virt exposed additional integration problems. These were addressed by:
cbcf574 fix(riscv): update test configuration
0f69f50 update(kao): add boot timeout
0fcda4b update(kao): use supported RISC-V QEMU
e17c3ab update(kao): report setup failures
80c2f98 update(kao): enable RISC-V Svpbmt
dea359b fix(riscv): build AIA test guest

with corresponding Bao Kao commits:
e2f3659 fix(riscv): enable AIA in QEMU
1334fec fix(logger): time out stalled boots
71d0579 fix(riscv): use supported QEMU version
f72572a fix(setup): report command failures
ad50039 fix(riscv): enable Svpbmt in QEMU

[miguelafsilva5]

Are all tests compiled and executed for all platforms? Let's say I want to develop and test a specific feature from a certain platform, how do I implement such tests? Can they be excluded from some architectures? Let's use cache coloring as an example. I might be reasonable to test if the coloring is working, but some platforms do not use caches.

[miguelafsilva5]

I was testing creating tests. And commenting a test does not compile.

Example:

/*
BAO_TEST(MYTESTS, BOOT, BAREMETAL,
    "Just a simple test to check that the test framework is working")
{
    if (cpu_is_master()) 
        TESTF_PASS("Test framework is working!\n");
}
*/

[josecm]

Using the TC4DX as an example (not sure if this is the same for other platforms). I believe kao assumes the board is always connected via ttyUSB0. I believe this is an hardcoded value. Maybe we can solve this by passing the port via command line? i.e. PORT=ttyUSB0. while leaving USB0 as default.

I think this is a very important point. Each platform driver must be behind an abstract interface (class?) and each platform implements its access details. Is this the current architecture?

[Diogo21Costa]

Using the TC4DX as an example (not sure if this is the same for other platforms). I believe kao assumes the board is always connected via ttyUSB0. I believe this is an hardcoded value. Maybe we can solve this by passing the port via command line? i.e. PORT=ttyUSB0. while leaving USB0 as default.

I think this is a very important point. Each platform driver must be behind an abstract interface (class?) and each platform implements its access details. Is this the current architecture?

Using the TC4DX as an example (not sure if this is the same for other platforms). I believe kao assumes the board is always connected via ttyUSB0. I believe this is an hardcoded value. Maybe we can solve this by passing the port via command line? i.e. PORT=ttyUSB0. while leaving USB0 as default.

I think this is a very important point. Each platform driver must be behind an abstract interface (class?) and each platform implements its access details. Is this the current architecture?

Platform-specific operations are already encapsulated in individual platform classes, and the Kao runner accesses them through a common set of methods, such as setup_platform(), build_firmware(), get_serial_ports(), and launch_test(). You are correct that each physical platform currently defines a hard-coded default serial device. To address this, I introduced an override through the --serial-port argument. When provided, this argument overrides the UART port defined by the selected platform; when omitted, Kao continues to use that platform’s existing default port. For example:
python3 src/kao.py -p tc4dx --serial-port /dev/ttyUSB1 ...
Thus, TC4DX still defaults to /dev/ttyUSB0, while users can select a different port without changing the platform driver.

This update introduces changes in the bao-kao repo only (bao-project/bao-kao@15006f7).

[Diogo21Costa]

Are all tests compiled and executed for all platforms? Let's say I want to develop and test a specific feature from a certain platform, how do I implement such tests? Can they be excluded from some architectures? Let's use cache coloring as an example. I might be reasonable to test if the coloring is working, but some platforms do not use caches.

A Kao invocation targets one platform only. Kao does not automatically execute the test suite across every platform; CI must invoke Kao separately for each target. To add a test, the author defines it using BAO_TEST(...) in a C source file and provides the corresponding setup configuration for the target platform under tests/tests/configs/<setup>/<platform>.yaml.
All test C source files are currently compiled into the guest, regardless of the selection. However, only the tests selected with -t are called and executed. Tests can be manually excluded from a particular invocation using --test-exclude.

There is currently no declarative mechanism for a test author to specify supported platforms, architectures, or required capabilities. Therefore, a cache-coloring test cannot currently be marked to automatically skip platforms without cache-coloring support; each platform invocation or CI matrix entry must explicitly include or exclude it.
Supporting this automatically would require introducing platform or capability requirements in the test metadata and filtering tests before building and executing them.

[Diogo21Costa]

I was testing creating tests. And commenting a test does not compile.

Example:

/*
BAO_TEST(MYTESTS, BOOT, BAREMETAL,
    "Just a simple test to check that the test framework is working")
{
    if (cpu_is_master()) 
        TESTF_PASS("Test framework is working!\n");
}
*/

You are correct. Kao was scanning the raw C source, so BAO_TEST definitions inside comments were still discovered and added to the generated test entry code, causing the build to fail. Fixed in bao-project/bao-kao@62ee65e

[miguelafsilva5]

Using the TC4DX as an example (not sure if this is the same for other platforms). I believe kao assumes the board is always connected via ttyUSB0. I believe this is an hardcoded value. Maybe we can solve this by passing the port via command line? i.e. PORT=ttyUSB0. while leaving USB0 as default.

I think this is a very important point. Each platform driver must be behind an abstract interface (class?) and each platform implements its access details. Is this the current architecture?

Using the TC4DX as an example (not sure if this is the same for other platforms). I believe kao assumes the board is always connected via ttyUSB0. I believe this is an hardcoded value. Maybe we can solve this by passing the port via command line? i.e. PORT=ttyUSB0. while leaving USB0 as default.

I think this is a very important point. Each platform driver must be behind an abstract interface (class?) and each platform implements its access details. Is this the current architecture?

Platform-specific operations are already encapsulated in individual platform classes, and the Kao runner accesses them through a common set of methods, such as setup_platform(), build_firmware(), get_serial_ports(), and launch_test(). You are correct that each physical platform currently defines a hard-coded default serial device. To address this, I introduced an override through the --serial-port argument. When provided, this argument overrides the UART port defined by the selected platform; when omitted, Kao continues to use that platform’s existing default port. For example: python3 src/kao.py -p tc4dx --serial-port /dev/ttyUSB1 ... Thus, TC4DX still defaults to /dev/ttyUSB0, while users can select a different port without changing the platform driver.

This update introduces changes in the bao-kao repo only (bao-project/bao-kao@15006f7).

@Diogo21Costa is this solution only available via python? Can I not overwrite the port using make?

If the make functionality is meant to be mainly used by the CI, I believe such an override still needs to be available. Imagine you have several boards connected to server. And for some reason, a board needs to be taken out to a live demo or something. The CI should be able to update the actions script easily to adapt to such conditions.

[miguelafsilva5]

Are all tests compiled and executed for all platforms? Let's say I want to develop and test a specific feature from a certain platform, how do I implement such tests? Can they be excluded from some architectures? Let's use cache coloring as an example. I might be reasonable to test if the coloring is working, but some platforms do not use caches.

A Kao invocation targets one platform only. Kao does not automatically execute the test suite across every platform; CI must invoke Kao separately for each target. To add a test, the author defines it using BAO_TEST(...) in a C source file and provides the corresponding setup configuration for the target platform under tests/tests/configs/<setup>/<platform>.yaml. All test C source files are currently compiled into the guest, regardless of the selection. However, only the tests selected with -t are called and executed. Tests can be manually excluded from a particular invocation using --test-exclude.

There is currently no declarative mechanism for a test author to specify supported platforms, architectures, or required capabilities. Therefore, a cache-coloring test cannot currently be marked to automatically skip platforms without cache-coloring support; each platform invocation or CI matrix entry must explicitly include or exclude it. Supporting this automatically would require introducing platform or capability requirements in the test metadata and filtering tests before building and executing them.

@Diogo21Costa my only gripe with this approach is that you are limiting a test developer from using platform-specific APIs. If I develop a test for TC4 using CSFR reads/writes, they won't compile for ARM or RISC-V. And then, you are putting a strain on the ARM and RISC-V developers to one-by-one exclude tests that don't compile from their builds.
Am I understanding this wrong?

[miguelafsilva5]

I was testing creating tests. And commenting a test does not compile.
Example:

/*
BAO_TEST(MYTESTS, BOOT, BAREMETAL,
    "Just a simple test to check that the test framework is working")
{
    if (cpu_is_master()) 
        TESTF_PASS("Test framework is working!\n");
}
*/

You are correct. Kao was scanning the raw C source, so BAO_TEST definitions inside comments were still discovered and added to the generated test entry code, causing the build to fail. Fixed in bao-project/bao-kao@62ee65e

Tested and working 👌

[josecm]

@Diogo21Costa my only gripe with this approach is that you are limiting a test developer from using platform-specific APIs. If I develop a test for TC4 using CSFR reads/writes, they won't compile for ARM or RISC-V. And then, you are putting a strain on the ARM and RISC-V developers to one-by-one exclude tests that don't compile from their builds. Am I understanding this wrong?

I think this makes a lot of sense. Each test should be couple with some description of it targets the hypervisor globally, a given architecture, or a given platform. And the framework could understand "oh! I only run this test on this platforms" or "I can't run this on this platform".

But maybe we can get the PR through without such feature and add it next.

[josecm]

@Diogo21Costa is this solution only available via python? Can I not overwrite the port using make?

If the make functionality is meant to be mainly used by the CI, I believe such an override still needs to be available. Imagine you have several boards connected to server. And for some reason, a board needs to be taken out to a live demo or something. The CI should be able to update the actions script easily to adapt to such conditions.

@miguelafsilva5 can you explain exactly what do you mean by "update the actions script"? You mean when the tty board for a given board changes name?

I kind of agree that the ci make rule should allow one to do this. I want to test and i need to override the tty port I write something like make test KAO_SERIAL=/dev/ttyUSB2, otherwise we need to call kao by hand no?

[miguelafsilva5]

@Diogo21Costa is this solution only available via python? Can I not overwrite the port using make?
If the make functionality is meant to be mainly used by the CI, I believe such an override still needs to be available. Imagine you have several boards connected to server. And for some reason, a board needs to be taken out to a live demo or something. The CI should be able to update the actions script easily to adapt to such conditions.

@miguelafsilva5 can you explain exactly what do you mean by "update the actions script"? You mean when the tty board for a given board changes name?

I kind of agree that the ci make rule should allow one to do this. I want to test and i need to override the tty port I write something like make test KAO_SERIAL=/dev/ttyUSB2, otherwise we need to call kao by hand no?

Yap. That's what I meant and that's what I propose.

[miguelafsilva5]

@Diogo21Costa my only gripe with this approach is that you are limiting a test developer from using platform-specific APIs. If I develop a test for TC4 using CSFR reads/writes, they won't compile for ARM or RISC-V. And then, you are putting a strain on the ARM and RISC-V developers to one-by-one exclude tests that don't compile from their builds. Am I understanding this wrong?

I think this makes a lot of sense. Each test should be couple with some description of it targets the hypervisor globally, a given architecture, or a given platform. And the framework could understand "oh! I only run this test on this platforms" or "I can't run this on this platform".

But maybe we can get the PR through without such feature and add it next.

We can opt for a simple solution in the meanwhile, like allowing folders inside tests/tests/src with platform names, and if the makefile finds a folder matching the PLATFORM=xxx name, it includes all the sources of that folder too.

But I'm also ok with moving forward with the PR and opening an issue

[miguelafsilva5]

@josecm @Diogo21Costa I think this might a bit out-of-scope, but may be worth addressing in some form of Test Guidelines.
Let's say we are testing multiple IRQ latencies. Our current test implementation do not disarm the interrupts they are testing, which leaves them interrupting other tests.
I think we have mention somewhere that each test is responsible to free the resources they use, once the test is finished.

[josecm]

@josecm @Diogo21Costa I think this might a bit out-of-scope, but may be worth addressing in some form of Test Guidelines. Let's say we are testing multiple IRQ latencies. Our current test implementation do not disarm the interrupts they are testing, which leaves them interrupting other tests. I think we have mention somewhere that each test is responsible to free the resources they use, once the test is finished.

This is a very good point 🤔 should we add some tests_clear_state() called by the harness before every tets, that brings the vm to a predefined state (ie, disables all interrupts, and we can think of other state it might require resetting...)

[miguelafsilva5]

@josecm @Diogo21Costa I think this might a bit out-of-scope, but may be worth addressing in some form of Test Guidelines. Let's say we are testing multiple IRQ latencies. Our current test implementation do not disarm the interrupts they are testing, which leaves them interrupting other tests. I think we have mention somewhere that each test is responsible to free the resources they use, once the test is finished.

This is a very good point 🤔 should we add some tests_clear_state() called by the harness before every tets, that brings the vm to a predefined state (ie, disables all interrupts, and we can think of other state it might require resetting...)

While that might be a good solution for the generic tests, how do you propose to implement a clear state for platform or architectural tests?
I mean, let's say we test a new function or peripheral in a certain platform. Do we have to add a new function or edit the clear state function to clear that specific peripheral? DMAs come to mind, multiple peripherals/tests might want to use the same channels. Same for IRQs. Are we clearing all the IRQ nodes?

Probably a better solution is to have the test developer be responsible for this and we as reviewers also be responsible for checking that before accepting any test.