DAOS-16311 test: Add C bindings for control API by mjmac · Pull Request #17550 · daos-stack/daos

mjmac · 2026-02-11T20:42:13Z

Use Go's c-shared-library mode to build a .so that
exposes a C API for the Go Control API. The primary
use case for this library is to replace the problematic
dmg helpers test library that uses fork/exec to allow
tests to drive dmg commands and does not play well
with UCX.

libdaos_control.so wraps the pool, system, storage,
server, and checker paths of lib/control, plus fault
injection hooks for the tests. tests_dmg_helpers.c is
rewritten on top of it, so daos_test and the ftest
cmocka helpers stop scraping dmg JSON. One side
effect of this change is that many operations complete
in milliseconds instead of seconds, and therefore some
tests needed to be updated to work properly with the new
timing.

Signed-off-by: Michael MacDonald github@macdonald.cx

github-actions · 2026-02-11T20:53:53Z

Ticket title is 'C bindings for DAOS control API'
Status is 'In Progress'
Labels: 'ci_2.6_daily,ci_2.6_provider,ci_master_daily,ci_master_provider'
https://daosio.atlassian.net/browse/DAOS-16311

daosbuild3 · 2026-02-11T22:20:52Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/2/testReport/

daosbuild3 · 2026-02-11T22:46:49Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/2/testReport/

daosbuild3 · 2026-02-11T23:56:25Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/2/testReport/

daosbuild3 · 2026-02-12T01:33:42Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/2/execution/node/1026/log

daosbuild3 · 2026-02-12T02:18:30Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/2/display/redirect

daosbuild3 · 2026-02-12T02:18:36Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/2/display/redirect

daosbuild3 · 2026-02-12T19:02:26Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/3/testReport/

daosbuild3 · 2026-02-12T19:19:32Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/3/testReport/

daosbuild3 · 2026-02-12T20:41:47Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/3/testReport/

daosbuild3 · 2026-02-22T19:24:43Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/4/testReport/

daosbuild3 · 2026-02-22T19:47:42Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/4/testReport/

daosbuild3 · 2026-02-22T21:09:02Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/4/testReport/

daosbuild3 · 2026-02-22T21:14:23Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/4/execution/node/1040/log

daosbuild3 · 2026-02-22T21:43:36Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/4/display/redirect

daosbuild3 · 2026-02-22T21:43:41Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/4/display/redirect

daosbuild3 · 2026-02-27T19:19:59Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/5/testReport/

daosbuild3 · 2026-02-27T19:20:52Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/5/testReport/

daosbuild3 · 2026-02-27T19:36:14Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/5/testReport/

daosbuild3 · 2026-02-27T20:55:20Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/6/testReport/

daosbuild3 · 2026-02-27T21:20:28Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/6/testReport/

daosbuild3 · 2026-02-27T23:12:43Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17550/6/testReport/

daosbuild3 · 2026-02-27T23:19:59Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/6/execution/node/1047/log

daosbuild3 · 2026-02-27T23:49:19Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/6/display/redirect

daosbuild3 · 2026-02-27T23:49:24Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17550/6/display/redirect

daosbuild3 · 2026-07-26T06:41:07Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/54/execution/node/1662/log

daosbuild3 · 2026-07-26T09:12:30Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/54/execution/node/1672/log

daosbuild3 · 2026-07-26T10:26:59Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/54/execution/node/1682/log

daosbuild3 · 2026-07-28T13:58:00Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/55/execution/node/1755/log

daosbuild3 · 2026-08-05T13:20:18Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/58/execution/node/829/log

Use Go's c-shared-library mode to build a .so that exposes a C API for the Go Control API. The initial use case for this library is to replace the problematic dmg helpers test library that uses fork/exec to allow tests to drive dmg commands. Skip-hw-medium-ucx-provider: false Features: cat_recov Signed-off-by: Michael MacDonald <github@macdonald.cx>

* daos_pool|daos_rebuild_ec: Don't escape self_heal prop value * control/server.go: Raw %v logging panics on nil; just remove it Signed-off-by: Michael MacDonald <github@macdonald.cx>

Update the cat_recov suite to work better with the new libdaos_control bindings, which operate more quickly and correctly than the old fork/exec dmg wrappers. - Check repair takes the CIA_ action value directly, where the old dmg CLI translated a selection index into the action. - The bindings answer queries in milliseconds where the fork/exec dmg helpers took 1-2s, which throws off some of the timing assumptions in the tests. Adjust to use a wall-clock budget instead of counting query iterations. - Pool-not-found now maps to DER_NONEXIST instead of DER_MISC, so cr_cleanup no longer needs to tolerate DER_MISC. - Switch SystemCheckDisable() to use Force: true on SystemStop(), matching the current system default and avoiding hangs on shutdown. - Misc correctness fixes in the gRPC handlers for SystemCheck(Enable|Disable). Skip-hw-medium-ucx-provider: false Features: cat_recov daos_core_test_nvme Signed-off-by: Michael MacDonald <github@macdonald.cx>

daosbuild3 · 2026-08-13T16:39:30Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/59/execution/node/1755/log

daosbuild3 · 2026-08-13T16:48:43Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17550/59/execution/node/1702/log

mjmac · 2026-08-27T18:58:44Z

The last few test runs have used the following pragmas:
Skip-hw-medium-ucx-provider: false
Features: cat_recov daos_core_test_nvme

The intent here was to really soak the verbs testing and the parts of the test framework that aren't normally run with the standard pr tag (those got run too, as Features is additive on top). I'm confident that the changes here don't introduce any new regressions, and the third commit fixes a few things.

Best I can tell, the two failures are pre-existing CR failures in the following tests:

test_stale_entry (check leader resumed instead of starting fresh)
cat_recov_core (fiddly query vs. destroy race)

At this point, I think it's time to try and get this landed.

daltonbohning

Just reviewed ftest and NLT code. Overall seems sane

daltonbohning · 2026-08-27T19:26:03Z

+            # Scope by pool UUID: resolved findings from prior checker instances survive
+            # disable/enable and reuse pool labels (DAOS-18773).
+            check_query_out = dmg_command.check_query(pool=pool_1.uuid)


Should we push an independent fix for this?

I'd also prefer an independent fix.

Dropped this as it was fixed by Kris' PR.

daltonbohning · 2026-08-27T19:32:55Z

+    control_so = join(conf['PREFIX'], 'lib64', 'libdaos_control.so')
+    if os.path.exists(control_so):


Should we error if it does not exist?

Yep, fixed.

kjacque

Can we pull the unrelated test and checker fixes out into separate PRs? Or are they just silencing bugs so you can get a better read on whether the control lib is working correctly?

kjacque · 2026-08-28T19:41:17Z

-		D_FREE(acl_str[i]);
-
-	D_FREE(acl_str);
+dmg_ctx_init(const char *cfg, uintptr_t *ctx)


This becomes sort of an intermediate step in this transformation, right? We should probably replace the name "dmg" in this filename and function names to "control" since it talks to the control lib instead.

Yeah, I just went ahead and renamed everything. Might as well do it now.

kjacque · 2026-08-28T19:50:29Z

+            # Scope by pool UUID: resolved findings from prior checker instances survive
+            # disable/enable and reuse pool labels (DAOS-18773).
+            check_query_out = dmg_command.check_query(pool=pool_1.uuid)


I'd also prefer an independent fix.

# Conflicts: # src/control/server/mgmt_check_test.go

* check_start_corner_case.py: drop the pool-scoped query workaround now that DAOS-18773 clears stale reports between cases * nlt/config.py: require a valgrind-tagged libdaos_control.so * control/c: cover whole-system stop/start failure paths * mgmt_check.go: restore the disableChecker error text Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

The helpers call libdaos_control directly; nothing on the path runs dmg any more, so the old prefix misdescribes what the tests depend on. Skip-hw-medium-ucx-provider: false Features: cat_recov daos_core_test_nvme Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

mjmac force-pushed the mjmac/DAOS-16311 branch from b1d5b89 to aa22df5 Compare February 11, 2026 20:49

mjmac force-pushed the mjmac/DAOS-16311 branch from aa22df5 to 9008458 Compare February 12, 2026 17:56

mjmac force-pushed the mjmac/DAOS-16311 branch from 9008458 to 249715f Compare February 22, 2026 18:36

mjmac force-pushed the mjmac/DAOS-16311 branch from 249715f to 4b56bdd Compare February 27, 2026 18:50

mjmac force-pushed the mjmac/DAOS-16311 branch from 4b56bdd to 096a1da Compare February 27, 2026 20:04

mjmac force-pushed the mjmac/DAOS-16311 branch from 799f81e to f373e5c Compare July 26, 2026 01:44

mjmac force-pushed the mjmac/DAOS-16311 branch from f373e5c to fd5ae50 Compare July 27, 2026 21:43

mjmac force-pushed the mjmac/DAOS-16311 branch from fd5ae50 to bc4072d Compare July 28, 2026 18:13

mjmac mentioned this pull request Jul 30, 2026

DAOS-623 test: instrument NLT fault-injection sweep hang #18756

Closed

mjmac force-pushed the mjmac/DAOS-16311 branch 2 times, most recently from 6a1ebe9 to 51863fd Compare August 5, 2026 12:27

mjmac added 3 commits August 12, 2026 14:47

DAOS-16311 test: misc fixes found during expanded testing

1822b0f

* daos_pool|daos_rebuild_ec: Don't escape self_heal prop value * control/server.go: Raw %v logging panics on nil; just remove it Signed-off-by: Michael MacDonald <github@macdonald.cx>

mjmac force-pushed the mjmac/DAOS-16311 branch from 51863fd to 9a7e563 Compare August 12, 2026 18:50

mjmac changed the title ~~DAOS-16311 control: Add C bindings for control API~~ DAOS-16311 test: Add C bindings for control API Aug 27, 2026

mjmac marked this pull request as ready for review August 27, 2026 19:08

mjmac requested review from a team as code owners August 27, 2026 19:08

daltonbohning reviewed Aug 27, 2026

View reviewed changes

kjacque reviewed Aug 28, 2026

View reviewed changes

mjmac added 3 commits November 6, 2026 22:37

Merge remote-tracking branch 'origin/master' into mjmac/DAOS-16311

5c0ab00

# Conflicts: # src/control/server/mgmt_check_test.go

		control_so = join(conf['PREFIX'], 'lib64', 'libdaos_control.so')
		if os.path.exists(control_so):

Conversation

mjmac commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Feb 11, 2026

Uh oh!

daosbuild3 commented Feb 11, 2026

Uh oh!

daosbuild3 commented Feb 11, 2026

Uh oh!

daosbuild3 commented Feb 11, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 12, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 22, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Feb 27, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 28, 2026

Uh oh!

daosbuild3 commented Aug 5, 2026

Uh oh!

daosbuild3 commented Aug 13, 2026

Uh oh!

daosbuild3 commented Aug 13, 2026

Uh oh!

mjmac commented Aug 27, 2026

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mjmac commented Feb 11, 2026 •

edited

Loading