DAOS-11169 control: decouple storage format from engine start by mjmac · Pull Request #19255 · daos-stack/daos

mjmac · 2026-10-12T21:36:44Z

Rework the default system bringup flow to decouple storage format
from engine startup. This separation enables easier automation by
blocking until each phase of deployment has finished with a clear
success or failure status.

With this commit, the following behavior is default:

dmg storage format -l [hostlist] does not start engines
dmg system initialize -l [hostlist] starts engines and reports
success or failure before exiting

The pre-3.0 storage format behavior may be enabled via the --start
flag. This commit also modifies the "Stopped" rank state such that
it may only be entered via dmg system stop, and a stopped rank
remains stopped until the admin starts it again.

Features: control server pool
Signed-off-by: Michael MacDonald mjmac@vectorsolutionsgroup.com

github-actions · 2026-10-12T21:37:19Z

Ticket title is 'Improve DAOS format/startup workflow with explicit system start'
Status is 'In Progress'
Labels: 'usability'
https://daosio.atlassian.net/browse/DAOS-11169

daosbuild3 · 2026-10-12T22:46:01Z

Test stage Test RPMs on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/2/execution/node/1291/log

daosbuild3 · 2026-10-12T22:46:45Z

Test stage Test RPMs on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/2/execution/node/1304/log

daosbuild3 · 2026-10-13T02:01:29Z

Test stage Test RPMs on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19255/4/display/redirect

daosbuild3 · 2026-10-13T15:47:24Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/5/execution/node/1475/log

daosbuild3 · 2026-10-14T14:13:25Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/6/execution/node/1434/log

daosbuild3 · 2026-10-14T16:10:33Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/6/execution/node/1721/log

daosbuild3 · 2026-10-15T03:16:29Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/6/execution/node/1860/log

daosbuild3 · 2026-10-15T12:26:26Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/6/execution/node/1917/log

daosbuild3 · 2026-10-17T06:36:46Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/7/execution/node/1852/log

daosbuild3 · 2026-10-17T16:59:47Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/8/execution/node/555/log

daosbuild3 · 2026-10-17T16:59:49Z

Test stage Build on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/8/execution/node/554/log

daosbuild3 · 2026-10-18T17:38:13Z

Test stage Functional on EL 9 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19255/9/testReport/

daosbuild3 · 2026-10-19T05:23:59Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/9/execution/node/1864/log

Engines will hold after format until released by dmg system initialize or extend. The server needs an RPC to release held engines, and the management service needs one that tells a rank whether to start. Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

Engines launch themselves after format, so a reboot restarts ranks the admin stopped on purpose and a fresh system starts before anyone asked. Hold them until dmg system initialize or extend releases them, or until the management service says the rank should restart; a dmg-stopped rank stays Stopped, out of the group map, until dmg system start. Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

Engines no longer launch themselves after format, so the harness, NLT and the tests that format storage must start them explicitly, and servers added to a running system join through dmg system extend. New tests cover the behaviour itself and the legacy configuration. Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

Features: control server pool Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

daosbuild3 · 2026-10-29T00:36:07Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19255/10/execution/node/1967/log

daosbuild3 · 2026-10-29T05:28:19Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19255/10/testReport/

mjmac · 2026-10-29T19:52:43Z

Known failures:

DAOS-15518 (dmg network scan/ucx)
DAOS-18951 (container destroy slowdown hits timeout)

tanabarr · 2026-11-03T16:04:20Z

oof this is a huge PR

tanabarr

first round of review comments

tanabarr · 2026-11-04T11:38:47Z

+// SystemExtendReq supplies system extend parameters.
+message SystemExtendReq
+{
+	string sys          = 1; // DAOS system name


nit: shares the same message fields as system init, embed shared message type

IMO this is fine as-is. Trying to combine things like that is a false economy that is undermined over time as things evolve and diverge, and we wind up with frankenmessages that have different fields used depending on the RPC.

tanabarr · 2026-11-04T11:41:10Z

 	uint32 daos_version = 7; // DAOS version that system supported.
 }

+// EnginePreflightReq asks what a server needs to know before it launches engines.


nit: clarity of comment could be improved, currently the check is just whether to start and why if not

The name of the method reflects the longer-term intent. You're right that currently it only checks to see if an engine should start, but the plan is to leverage this to rework the CR mode switch logic so that we don't have the weird start/join/restart/join dance that can lead to mode bugs (e.g. https://daosio.atlassian.net/browse/DAOS-18622).

tanabarr · 2026-11-04T22:31:27Z

+			continue
+		}
+		if rank, _ := ei.GetRank(); !rank.Equals(e.Rank) {
+			h.log.Noticef("instance %d: preflight result for rank %s ignored; now rank %s and asking again",


When would this rank update happen? does this case cover the initial assignment or later change? does the "asking again" statement refer to the next time to the next time runPreflight() gets called on the timer? Presumably the next time it's run the request gets sent with the new updated rank in the request?

The log message should be fixed. What this guards against is a race where a preflight request is made for rank X, and then the engine instance is erased/reformatted before the response comes back. A new preflight request will be made for the reformatted instance, so we can ignore this stale response.

daltonbohning · 2026-11-05T21:02:51Z

+        Returns:
+            tuple: the host as a NodeSet and its rank
+        """
+        host = NodeSet(self.server_managers[0].hosts.difference(self.mgmt_svc_replicas)[-1])


If you wanted to sprinkle some randomness

Suggested change

host = NodeSet(self.server_managers[0].hosts.difference(self.mgmt_svc_replicas)[-1])

host = NodeSet(

self.random.choice(self.server_managers[0].hosts.difference(self.mgmt_svc_replicas)))

Nice, added that in.

daltonbohning · 2026-11-05T21:05:53Z

+        Args:
+            host (NodeSet): host to check
+            seconds (int): seconds to keep checking
+        """
+        for _ in range(seconds):
+            if self.engine_running(host):
+                self.fail(f"An engine started on {host}")
+            time.sleep(1)


Nit - The run_remote / clush can sometimes take ~0.1 seconds or more depending on the system. So technically this is trying up to seconds TIMES, not necessarily 30 seconds. But not a big deal

Yeah, good point. I'll change it to have a proper deadline if I need to make other changes.

daltonbohning · 2026-11-05T21:07:28Z

+        if self.server_managers[0].check_rank_state([rank], [state], checks):
+            self.fail(f"Rank {rank} did not reach the {state} state")
+
+    def test_initialize(self):


These test_* names are global across the framework, so I'd recommend putting some context in there. Maybe one of these?

test_dmg_system_initialize test_dmg_system_init

Fixed, thanks.

daltonbohning · 2026-11-05T21:14:16Z

+        self.log_step('Scenario 2 - Kill a rank and verify it is not excluded')
+        self.server_managers[0].kill_ranks(ranks_x)


Why change the stop to a kill? Are we potentially missing test coverage?

This PR changes the Stopped state such that it can only be entered or exited by explicit admin action, and it's persistent. Some of the tests relied on stopped ranks eventually going into the Excluded state, but Stopped->Excluded is an invalid transition now, deliberately. In order for a rank to be Excluded, it has to have exited unexpectedly and been marked dead by SWIM.

daltonbohning · 2026-11-05T21:15:14Z

        telemetry_before = self.collect_telemetry(rank_count=rank_count)

-        # 2. Stop a rank.
+        # 2. Kill a rank.


Similar here. Why change the stop to a kill?

…plicit-startup

* Fix test names to reflect test_dmg_[cmd] idiom * Randomize the server used for testing engine restarts * Improve log message for stale preflight response Features: control server pool Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from 8623f37 to dbee278 Compare October 12, 2026 21:40

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from dbee278 to 51c54c6 Compare October 13, 2026 00:21

mjmac changed the title ~~mjmac/DAOS 11169 explicit startup~~ DAOS-11169 control: hold engines for an explicit system start Oct 13, 2026

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch 2 times, most recently from c5ffb49 to 0971682 Compare October 13, 2026 01:59

mjmac changed the title ~~DAOS-11169 control: hold engines for an explicit system start~~ DAOS-11169 control: decouple format from engine start Oct 13, 2026

mjmac changed the title ~~DAOS-11169 control: decouple format from engine start~~ DAOS-11169 control: decouple storage format from engine start Oct 13, 2026

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from 0971682 to 3420d29 Compare October 13, 2026 15:45

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from 3420d29 to 8ec4ada Compare October 15, 2026 16:45

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from 8ec4ada to 4ef6d2f Compare October 17, 2026 16:54

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from 4ef6d2f to d459450 Compare October 17, 2026 19:04

mjmac added 4 commits October 27, 2026 11:59

DAOS-11169 doc: describe explicit system startup

90e6366

Features: control server pool Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

mjmac force-pushed the mjmac/DAOS-11169-explicit-startup branch from d459450 to 90e6366 Compare October 27, 2026 16:18

mjmac marked this pull request as ready for review October 27, 2026 19:58

mjmac requested review from a team as code owners October 27, 2026 19:58

mjmac requested review from kjacque and tanabarr and removed request for a team October 27, 2026 19:58

tanabarr reviewed Nov 4, 2026

View reviewed changes

daltonbohning reviewed Nov 5, 2026

View reviewed changes

mjmac added 2 commits November 7, 2026 08:51

Merge remote-tracking branch 'origin/master' into mjmac/DAOS-11169-ex…

5ccb4f3

…plicit-startup

Address review feedback

7d3608c

* Fix test names to reflect test_dmg_[cmd] idiom * Randomize the server used for testing engine restarts * Improve log message for stale preflight response Features: control server pool Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>

	host = NodeSet(self.server_managers[0].hosts.difference(self.mgmt_svc_replicas)[-1])
	host = NodeSet(
	self.random.choice(self.server_managers[0].hosts.difference(self.mgmt_svc_replicas)))

		self.log_step('Scenario 2 - Kill a rank and verify it is not excluded')
		self.server_managers[0].kill_ranks(ranks_x)

Conversation

mjmac commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 13, 2026

Uh oh!

daosbuild3 commented Oct 13, 2026

Uh oh!

daosbuild3 commented Oct 14, 2026

Uh oh!

daosbuild3 commented Oct 14, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 17, 2026

Uh oh!

daosbuild3 commented Oct 17, 2026

Uh oh!

daosbuild3 commented Oct 17, 2026

Uh oh!

daosbuild3 commented Oct 18, 2026

Uh oh!

daosbuild3 commented Oct 19, 2026

Uh oh!

daosbuild3 commented Oct 29, 2026

Uh oh!

daosbuild3 commented Oct 29, 2026

Uh oh!

mjmac commented Oct 29, 2026

Uh oh!

tanabarr commented Nov 3, 2026

Uh oh!

tanabarr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

mjmac commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading