[RFD] Node Lifecycle State Machine

[alexlovelltroy]

Context

Many provisioners and other HPC management tools implement a state machine for nodes. As nodes transition in and out of service and through firmware updates, the tooling provides a shared understanding of node state and which steps are required to bring it back into service. Different tools have different sets of states and different transitions between those states that are available. Sites have come to rely on their own interpretation of various indicators. Given OpenCHAMI's emphasis on composability, we should consider two questions. First, should OpenCHAMI support a state machine for nodes or expect sites to have their own. Second, if the community decides to support the state machine, should it be internal or external.

Proposal

OpenCHAMI needs a mechanism to manage the lifecycle of HPC nodes as they transition through various states such as in_service, maintenance, rma, firmware_update, etc.

Two architectural options are under consideration:

1. Embed the state machine inside the Inventory Service (SMD)
2. Use an external state machine through a workflow engine

Each approach aims to support:

• Node state transitions
• Integration with site-specific tooling (e.g., firmware checks, diagnostics)
• Notifications or side effects (e.g., webhook calls, logging, tagging)

The goal of this RFD is to evaluate both strategies and support a clear, maintainable design choice.

---

Option 1: Embedded State Machine in Inventory Server

The state transition logic lives inside the inventory service (SMD), and node objects include a state field. Transitions are enforced by the inventory API. Notifications to other systems are sent via webhooks.

Example: State Machine Logic (Go)

type NodeState string

const (
    StateInService      NodeState = "in_service"
    StateMaintenance    NodeState = "maintenance"
    StateFirmwareUpdate NodeState = "firmware_update"
)

var validTransitions = map[NodeState][]NodeState{
    StateInService:      {StateMaintenance},
    StateMaintenance:    {StateFirmwareUpdate, StateInService},
    StateFirmwareUpdate: {StateInService},
}

func isValidTransition(from, to NodeState) bool {
    for _, valid := range validTransitions[from] {
        if valid == to {
            return true
        }
    }
    return false
}

### Applying a State Transition

```go
func UpdateNodeState(nodeID string, newState NodeState) error {
    node, err := db.GetNode(nodeID)
    if err != nil {
        return err
    }

    if !isValidTransition(node.State, newState) {
        return fmt.Errorf("invalid state transition from %s to %s", node.State, newState)
    }

    node.State = newState
    err = db.SaveNode(node)
    if err != nil {
        return err
    }

    // Send webhook for notification
    go notifyWebhooks(node)

    return nil
}

Benefits

• Simpler deployment: fewer moving parts
• Easier to understand and trace single-system logic
• REST API directly reflects authoritative state

Drawbacks

• Tightly couples state logic to inventory system
• Difficult to customize transitions per site
• Harder to add async or long-lived logic (e.g., firmware retry, RMA waiting)
• Operational complexity increases with embedded hooks

Option 2: External Workflow Engine for State Management

In this model, the inventory system is decoupled from the state transition logic. Instead, an external workflow engine (e.g., Temporal) manages the node lifecycle as a series of state transitions. The inventory system records the current state and also maintains a trail of all past transitions, enabling full reconstruction of a node’s lifecycle history.

Workflow Engine Responsibilities

• Own and enforce the valid transitions between states
• Perform asynchronous tasks (e.g., firmware checks, diagnostics, RMA wait)
• Call back to the inventory service to:
  • Update the node’s current state
  • Append a structured entry to the state transition log

Inventory System Responsibilities

• Expose a REST API for getting and updating node metadata
• Store:
  • current_state (e.g., in_service)
  • state_history: a chronological list of transitions with metadata (timestamps, initiator, reason)

Example transition log entry:

{
  "from": "maintenance",
  "to": "firmware_update",
  "timestamp": "2025-06-18T16:44:00Z",
  "initiator": "temporal:ReturnToServiceWorkflow",
  "note": "Automated firmware validation triggered"
}

Benefits

• Supports site-specific lifecycle requirements without bloating inventory logic
• Enables asynchronous and long-running transitions (e.g., RMA wait)
• Provides an auditable history of state changes
• Easier to evolve workflows independently of the inventory API
• External workflow systems have a broader community of support than an internal state machine would. Likely to be more customizable as well.

Drawbacks

• Adds operational complexity (deployment and monitoring of workflow engine)
• Requires consistency between workflow engine and inventory state
• Developers must manage state synchronization explicitly (e.g., retries on failed updates)

References:

• https://pages.temporal.io/rs/250-WIU-007/images/whitepaper-state-machines-simplified.pdf

[alexlovelltroy]

A third option to consider is an internal state machine that is not particularly flexible, but that can be disabled to allow an external state machine to function instead.

[jsollom-hpe]

LANL investigated a distributed state engine framework called Kraken.

[jsollom-hpe]

I do not think the state machine logic should be part of the Inventory service. The Inventory service should be updated with the state and can record its history, but nothing more than that. Anything beyond that unnecessarily entangles the inventory with decisions about the state of nodes.

[alexlovelltroy]

LANL investigated a distributed state engine framework called Kraken.

Kraken was truly innovative and we learned a lot from it. In my opinion, the administrative challenges associated with distributed state as deployed with kraken were too significant to pursue for OpenCHAMI.

[alexlovelltroy]

I do not think the state machine logic should be part of the Inventory service. The Inventory service should be updated with the state and can record its history, but nothing more than that. Anything beyond that unnecessarily entangles the inventory with decisions about the state of nodes.

I agree with you and generally favor one or more external state machines.

However, is there value in having the inventory service understand the external state machine(s) in order to report to users more detailed status information?

[jsollom-hpe]

However, is there value in having the inventory service understand the external state machine(s) in order to report to users more detailed status information?

No, I don't think having it understand the state is useful. It can log the state so that it is easy for system admins to access. If it is not taking any action based on the state, it does not need to understand it.

[jsollom-hpe]

Here are some things I learned designing and supporting the Boot Orchestration Service (BOS) in the Cray System Management (CSM) software stack.

Trade-offs between declarative vs. imperative state management

BOS had an imperative mode (default mode) and a declarative mode. The declarative mode was never fully tested, so it should be considered beta. When I say a declarative mode, I mean that BOS would continuously monitor the state of nodes and try to ensure they were in the desired state. When I say imperative, it meant BOS would attempt to move the node to the desired state and then it would quit, so it was more of a 'one-shot' approach. However, even in imperative mode, there were retries to handle failures. In imperative mode, BOS did not continuously monitor the nodes’ state.

The imperative mode worked well enough that we did not feel a pressing need to fully test the declarative mode and put it into service. However, we felt customers might want such a mode which is why we implemented it in the first place. The idea was that BOS could handle deviations from the desired state and minimize system administrator monitoring and maitenance.

Challenges with "unexpected" automated actions (e.g., nodes rebooting during debugging)

In imperative mode, BOS would retry operations when it experienced failures. For example, if a node failed to boot, BOS would shut it down and boot it again up to the allowed number of retries. It was during these retries that HPE would sometimes receive complaints because BOS had rebooted a node on which a system administrator was trying to debug the failure. The way to prevent this issue was to tell BOS to stop operating on the node. This was simple and straight-forward enough, and it could be done through the Cray CLI. However, the need to disable the node in BOS may not have been intuitive.

Scalability of heartbeat/state-reporting mechanisms

BOS provided a state-reporting mechanism where a BOS client that was part of the boot image on the compute node would check back in at a configurable interval. By default, the interval was every four hours. Thus, the node would boot up, the ‘bos-reporter’ client would update the node’s booted state, and then it would check in every four hours afterwards. Thus, the state-reporting was not very heavy-weight because it was infrequent, and the message payload was tiny.

Logging fragmentation solutions (e.g., session IDs across microservices)

In BOS Version 1, BOS launched a Kubernetes job that ran in a pod. The pod’s logs were self-contained, and it made it easy to track the entire boot session.

In BOS Version 2, HPE rearchitected BOS into multiple operators that each had their own log. Because each operator handled part of a boot session, the boot session information was scattered across each of the operators’ logs. This made it hard to track. HPE did not provide a trace-ID which likely would have helped solve this problem.

[jsl-hpe]

I would just like to draw attention to a few issues we have encountered while dealing with the concepts of recorded state in CSM, and why we chose a few of the things we did.

First, we attempt to keep state stored on a per service basis, for a few different reasons. The first is, that multiple services interact with these records, both in a their query and set nature, and unless all services agree on the purpose of a given field, it is difficult to ensure consistent behavior. You may recall an antiquated field in the inventory service called 'ready'. Well, as it turns out, ready means many different things depending on the context of what you want to happen. In some senses, this free form field attempted to provide users a sense of utility for a given piece of hardware, however, having a node simply powered on is not sufficient to indicate that the environment is fully provisioned. Because various power services had different behavior when it came to setting and enforcing their own behavior for this field, the field quickly became a chaotic mess. Similarly, just because a component is disabled in BOS doesn't mean that other hardware management APIs shouldn't work. Having the granularity to disable a component from provisioning allows other services, like a firmware action service, power capping, or even power locking features to still work. It may seem like this nuance is trivial, but when it comes to the implementation, the behaviors that they control are highly important.

Second, we keep state records with the APIs that implement behavior around them, for reasons of composability. This means that should someone want to remove any of our APIs, the only state fields that are affected are those that are purposefully replaced. As it turns out, when state is consolidated to the APIs that implement their behavior, the records turn out way cleaner. Generally speaking, a power management API shouldn't particularly care what WLM is running on a given system, and a configuration management system shouldn't bother about any records that pertain to the firmware revision version that has been flashed to a piece of hardware. Because this state is paired to the services that implements the request, it provides utility to specifically those clients that find that information germane. Said another way, when we implement an operator that drives to a particular state, all of its behavior is controlled via a central point and typically at most requires one CRUD operation to find the necessary information in question.

Finally, and perhaps most importantly, we do this for means of performance. The configuration framework service in CSM applies a known configuration against a node or an image within a chroot jail. We implemented operators that examine which components need to be configured, and store this information in the API. This allows us not only to have a protected place where every operator can agree on what needs to happen, but also eliminates the need for a more invasive query operation which talks with each running node in order to ascertain state. This allows us to minimize expensive queries that touch or interrupt compute hardware for interrogation, as well, allows us to quickly determine what needs to be done to the system from a central location. Only updating these states when we know they have changed is a boon.

Reflective state of managed hardware is often employed in this fashion, both for requests of state transfer as well as actual state of a node. This is useful, because often times requests cannot be honored for thread locked or transient errors. If BOS wants to provision a node using a specific kernel/initrd/rootfs, but the node is taken offline for administrative actions, we still believe there is merit in the request, so we record the desired state and then apply it when those pieces of hardware have finished their administrative actions. When we record the desired state, it further complicates the meaning of individual state fields, which further enforces our need to keep those records separate and unique to operators which needs to act upon them.

[alexlovelltroy]

I think the cleanest path forward here is to lean into a CQRS + Event Sourcing approach for the smd inventory layer, and let something outside of smd own the actual state machine for nodes.

In this model, smd remains the system of record. It is authoritative for what hardware exists and static/dynamic attributes. It exposes APIs optimized for reading and updating current state. All mutations to inventory data should also emit events that form an append-only log of changes that can be replayed to rebuild state. The smd notification service may serve as the basis for these events, but I'd like to see us explore options with lower infrastructure and development costs. Perhaps the Outbox Pattern. Parquet files in an S3 Bucket? Kafka/NATS/Webhooks?

A separate workflow engine, starting with go-workflows and leaving a migration path to Temporal if we need more scale, would consume events and coordinate node state transitions across multiple microservices. That engine can handle long-running, multi-step processes like changing boot targets, waiting for log signals, performing power cycles, and verifying health, without bloating the smd codebase. It also gives us observability into workflows, retries, and recovery paths, while keeping smd focused and simple.

This separation lets us:

• Keep smd small, predictable, and easy to deploy at the “minimum infrastructure” tier.
• Scale orchestration independently, swapping workflow engines if needed without rewriting domain logic.
• Build richer state transition logic without coupling it to the CRUD layer.

To illustrate what workflow engine code can look like, I created this trivial example to show what rolling out reboots of nodes might look like following this model.

// workflow functions always take a cancelable context as a first argument
// in this case, we also include the BootSpec representing any kernel, initrd, image, cloud-init changes
func Rollout(ctx context.Context, c *Client, nodes []string, spec BootSpec) error {
	// 1) Update boot spec for all nodes (sequential or batched; kept simple here)
	for _, n := range nodes {
		if err := c.UpdateBootSpec(ctx, n, spec); err != nil {
			return fmt.Errorf("node %s: %w", n, err)
		}
	}

	// 2) Fan out: power cycle + wait for phone-home (retry power once if needed)
	eg, ctx := errgroup.WithContext(ctx)
	eg.SetLimit(25) // limit concurrency if desired

	for _, nodeID := range nodes {
		n := nodeID
                 //  Taking advantage of go concurrency to run a separate microthread for each node in the rollout.
                 //  Workflow state is traceable even within the go func which makes it possible to see which functions
                 //  succeeded and failed independently and infer the state of each node. 
		eg.Go(func() error {
			// First power cycle (this is a call to power-control)
			if err := c.PowerCycle(ctx, n); err != nil {
				return fmt.Errorf("power cycle (%s): %w", n, err)
			}
                         // WaitForPhoneHome would be implemented elsewhere to tail logs and/or wait for Events
                         // that indicate a node has made it through boot and called the phone-home API from cloud-init
			if ok := c.WaitForPhoneHome(ctx, n, 10*time.Minute); ok {
				return nil
			}

			// Retry once (extensible for multiple retries)
			if err := c.PowerCycle(ctx, n); err != nil {
				return fmt.Errorf("retry power cycle (%s): %w", n, err)
			}
			if ok := c.WaitForPhoneHome(ctx, n, 10*time.Minute); !ok {
				return fmt.Errorf("phone-home timeout after retry (%s)", n)
			}
			return nil
		})
	}
	return eg.Wait()
}