DAOS-18690 vos: fix engine crash when extend committed DTX table by Nasf-Fan · Pull Request #18671 · daos-stack/daos

Nasf-Fan · 2026-07-15T07:03:09Z

PR#18141 (b36dd7b) introduced some regression:

When the pool is physically full (ENOSPC), extending committed DTX table for punch/update commit may crash the whole engine with SIGABRT. That is because the backend TX logic (such as PMDK) will abort the TX by default if hit failure when allocation. Then subsequent operation (such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table()) against such aborted backend TX will trigger assertion inside backend TX logic.

The patch replaces umem_zalloc() with umem_alloc_verb() for extending committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify backend TX logic to return ENOSPC without aborted related TX. Similar enhancement is also added for tx_add_ptr related logic.

New unit tests (UMEM010/BMEM016/BMEM024) for that.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-07-15T07:03:29Z

Ticket title is 'Aurora daos_user: SCM single target ran out of space (min:0 B) and not able to finish GC. '
Status is 'In Review'
Labels: 'request_for_2.8.1'
https://daosio.atlassian.net/browse/DAOS-18690

daosbuild3 · 2026-07-15T08:12:48Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/2/testReport/

daosbuild3 · 2026-07-15T08:24:24Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/2/testReport/

hgichon

Thanks @Nasf-Fan — cross-checked against #18612 and everything landed: flag consistency, full failure_behavior unwind (no dangling refs), and the commit path (vos_dtx_commit_internal) is fully NO_ABORT-protected, so the crash can't recur. Nice that UMEM010 now covers the xadd_ptr path too. LGTM.

Two non-blocking checks:

vos_dtx_extend_act_table() / vos_dtx_set_flags_one() reverted to plain umem_tx_add_ptr() (abort on failure) — intentional for those non-commit paths?
UMEM010 runs under setup_pmem only; the reported crash was on md-on-SSD (DAV). Same coverage as #18612, just flagging in case a bmem variant is wanted.

Nasf-Fan · 2026-07-16T04:02:42Z

Thanks @Nasf-Fan — cross-checked against #18612 and everything landed: flag consistency, full failure_behavior unwind (no dangling refs), and the commit path (vos_dtx_commit_internal) is fully NO_ABORT-protected, so the crash can't recur. Nice that UMEM010 now covers the xadd_ptr path too. LGTM.

Two non-blocking checks:

vos_dtx_extend_act_table() / vos_dtx_set_flags_one() reverted to plain umem_tx_add_ptr() (abort on failure) — intentional for those non-commit paths?

vos_dtx_add_ptr() is used to handle space pressure when we try to release some space (directly or indirectly), such as DTX aggregation or DTX commit. But vos_dtx_extend_act_table() is during update/punch processing that will consume more space, even if we use vos_dtx_add_ptr(), related modification itself will also consume more space and maybe out of space when space pressure. So they are not the candidate users for vos_dtx_add_ptr().

UMEM010 runs under setup_pmem only; the reported crash was on md-on-SSD (DAV). Same coverage as DAOS-18690 vos: fix engine abort in DTX commit blob fallback on ENOSPC #18612, just flagging in case a bmem variant is wanted.

I will add new test cases to cover bmem and bmem_v2.

daosbuild3 · 2026-07-16T04:56:33Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/4/testReport/

daosbuild3 · 2026-07-16T05:14:22Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/4/testReport/

daosbuild3 · 2026-07-17T05:04:24Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18671/4/execution/node/1773/log

daosbuild3 · 2026-07-17T05:04:28Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18671/4/execution/node/1763/log

PR#18141 (b36dd7b) introduced some regression: When the pool is physically full (ENOSPC), extending committed DTX table for punch/update commit may crash the whole engine with SIGABRT. That is because the backend TX logic (such as PMDK) will abort the TX by default if hit failure when allocation. Then subsequent operation (such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table()) against such aborted backend TX will trigger assertion inside backend TX logic. The patch replaces umem_zalloc() with umem_alloc_verb() for extending committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify backend TX logic to return ENOSPC without aborted related TX. Similar enhancement is also added for tx_add_ptr related logic. New unit tests (UMEM010/BMEM016/BMEM024) for that. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: hgichon <hgichon@gmail.com>

knard38

LGTM

knard38 · 2026-07-21T14:42:39Z

 */
 int utest_check_mem_initial_status(struct utest_context *utx);

+/** Check transaction failure with NO_ABROT flag


Suggested change

/** Check transaction failure with NO_ABROT flag

/** Check transaction failure with NO_ABORT flag

Nasf-Fan · 2026-07-22T13:12:16Z

Ping reviewers, thanks!

hgichon · 2026-07-23T16:18:30Z

Apologies for the very late confirmation on this — thank you for reworking the fix.

I verified this patch (feb6d69) against the original DAOS-18690 scenario we hit during our project-quota work:

Cherry-picked cleanly onto our exastor feature/proj-quota branch (only src/common/mem.c needed a trivial adaptation for our older POBJ_XALLOC_NO_ABORT → your POBJ_FLAG_TX_NO_ABORT naming; no functional change).
Built and ran the VOS unit tests: UMEM010: Test TX failure with NO_ABORT on pmem passes, i.e. a failed NO_ABORT allocation now leaves the TX alive and subsequent alloc/tx_add_ptr/commit succeed (this crashed/aborted the engine before the fix). All 10 umem tests green.
Full regression of our projid-quota feature on top of this fix: container proj_quota set-prop, projid xattr set/get, directory inheritance, over-limit enforcement, and unlimited isolation all pass, with no engine SIGABRT across the run.

So this resolves the DTX-commit-under-ENOSPC engine crash for us, and the umem_alloc_verb + UMEM_FLAG_NO_ABORT approach (with umem_tx_xadd_ptr in vos_dtx_add_ptr) is a clean superset of what we had locally. LGTM — thanks again.

hgichon

Verified against the original DAOS-18690 scenario (UMEM010 NO_ABORT regression + full projid-quota regression, no engine abort) — see comment above. LGTM.

The reusing behavior is equal to (DTX) aggregating one committed DTX blob from victim container. So need to refresh relatred ::cd_newest_aggregated. Signed-off-by: Fan Yong <fan.yong@hpe.com>

Nasf-Fan · 2026-08-09T02:38:34Z

Ping reviewers, thanks!

daltonbohning · 2026-08-11T15:03:35Z

This still needs a second approval before landing.

Resolve merge conflict. Signed-off-by: Fan Yong <fan.yong@hpe.com>

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch from 23eab47 to 71c21bd Compare July 15, 2026 07:32

Nasf-Fan mentioned this pull request Jul 15, 2026

DAOS-18690 vos: fix engine abort in DTX commit blob fallback on ENOSPC #18612

Closed

hgichon reviewed Jul 15, 2026

View reviewed changes

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch 2 times, most recently from 9619ce3 to 3b39604 Compare July 16, 2026 04:19

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch from 3b39604 to 7788fea Compare July 17, 2026 05:02

Nasf-Fan marked this pull request as ready for review July 18, 2026 01:47

Nasf-Fan requested review from a team July 18, 2026 01:47

Nasf-Fan requested review from a team as code owners July 18, 2026 01:47

Nasf-Fan requested review from hgichon, janekmi and knard38 and removed request for hgichon July 18, 2026 01:47

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch from 7788fea to feb6d69 Compare July 21, 2026 04:27

knard38 reviewed Jul 21, 2026

View reviewed changes

knard38 previously approved these changes Jul 21, 2026

View reviewed changes

hgichon approved these changes Jul 23, 2026

View reviewed changes

Nasf-Fan requested a review from a team July 26, 2026 04:39

DOAS-18690 vos: refresh cd_newest_aggregated

c9e5cc3

The reusing behavior is equal to (DTX) aggregating one committed DTX blob from victim container. So need to refresh relatred ::cd_newest_aggregated. Signed-off-by: Fan Yong <fan.yong@hpe.com>

Nasf-Fan dismissed knard38’s stale review via c9e5cc3 July 26, 2026 14:37

Merge branch 'master' into Nasf-Fan/DAOS-18690_5

4e513ab

daltonbohning removed the request for review from a team July 28, 2026 16:35

Nasf-Fan requested review from gnailzenh and knard38 July 29, 2026 03:42

hgichon mentioned this pull request Aug 6, 2026

DAOS-XXXXX vos: keep TX alive when DTX commit blob alloc hits ENOSPC #18803

Closed

knard38 previously approved these changes Aug 10, 2026

View reviewed changes

Nasf-Fan requested a review from a team August 11, 2026 06:49

daltonbohning removed the request for review from a team August 11, 2026 15:03

Merge branch 'master' into Nasf-Fan/DAOS-18690_5

d9019ec

Resolve merge conflict. Signed-off-by: Fan Yong <fan.yong@hpe.com>

Nasf-Fan dismissed knard38’s stale review via d9019ec August 19, 2026 05:21

knard38 approved these changes Aug 19, 2026

View reviewed changes

gnailzenh approved these changes Aug 26, 2026

View reviewed changes

gnailzenh merged commit 434fec6 into master Aug 26, 2026
46 of 49 checks passed

gnailzenh deleted the Nasf-Fan/DAOS-18690_5 branch August 26, 2026 08:52

	/** Check transaction failure with NO_ABROT flag
	/** Check transaction failure with NO_ABORT flag

Conversation

Nasf-Fan commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jul 15, 2026

Uh oh!

daosbuild3 commented Jul 15, 2026

Uh oh!

hgichon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 17, 2026

Uh oh!

daosbuild3 commented Jul 17, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

knard38 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jul 22, 2026

Uh oh!

hgichon commented Jul 23, 2026

Uh oh!

hgichon left a comment

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Aug 9, 2026

Uh oh!

daltonbohning commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

Nasf-Fan commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading

hgichon left a comment •

edited

Loading