Skip to content

AGENTS.md

Latest commit

 

History

History
191 lines (145 loc) · 8.36 KB

AGENTS.md

File metadata and controls

191 lines (145 loc) · 8.36 KB

AGENTS.md

This file provides guidance to AI code assistants when working with code in this repository.

Overview

The Cluster Version Operator (CVO) is one of the Cluster Operators that run in every OpenShift cluster. CVO consumes an artifact called a "release payload image," which represents a specific version of OpenShift. The release payload image contains the resource manifests necessary for the cluster to function, including manifests for all Cluster Operators. CVO reconciles the resources within the cluster to match the manifests in the release payload image, implementing cluster upgrades by this reconciliation process.

CVO continuously monitors for available updates from the OpenShift Update Service (OSUS) and orchestrates the systematic rollout of new versions. When provided a release payload image for a newer OpenShift version, CVO reconciles all Cluster Operators to their updated versions, and those Cluster Operators similarly update their operands, completing the cluster upgrade.

Building and Testing

Build commands

# Build the CVO binary
make build                # Runs hack/build-go.sh, outputs to _output/

# Build the CVO container image
./hack/build-image.sh     # Creates local image with git version tag

# Push development image to a registry
REPO=quay.io/yourname ./hack/push-image.sh

# Format code
make format              # Run go fmt on all packages

Testing commands

# Unit tests
make test               # Runs gotestsum with JUnit XML output to _output/junit.xml

# Single package test
gotestsum --packages="./pkg/cvo"

# Integration tests (requires KUBECONFIG with admin credentials for a disposable cluster)
make integration-test   # Runs tests with TEST_INTEGRATION=1 against real cluster

# Update test metadata (for tests in test/ directory)
make update

# Verification
make verify             # Runs verify-yaml and verify-update
make verify-yaml        # Validates YAML manifests
make verify-update      # Ensures generated files are up-to-date

Architecture

Core Components

Main Operator Loop (pkg/cvo/cvo.go)

  • The Operator struct is the central controller
  • Watches the ClusterVersion resource in cluster
  • Implements a reconciliation loop that is the single writer of ClusterVersion status
  • Uses a work queue pattern with 15 max retries for failed operations
  • Periodically checks for updates from configured upstream server
  • Maintains conditions: Progressing, Available, Failing, Upgradeable, RetrievedUpdates

SyncWorker (pkg/cvo/sync_worker.go)

  • Abstracts payload synchronization via ConfigSyncWorker interface
  • Manages the actual application of manifests from release payload images
  • Handles payload retrieval, verification, and application
  • Uses rate limiting and retry logic for resilient updates
  • Reports status back to main operator via SyncWorkerStatus channel

Payload Management (pkg/payload/)

  • State enum defines update modes: UpdatingPayload, ReconcilingPayload, InitializingPayload, PrecreatingPayload
  • UpdatingPayload: Conservative ordering when transitioning versions, errors block dependent operators
  • ReconcilingPayload: Maintains current state, recreates resources without strict ordering
  • InitializingPayload: First-time deployment, fast progress with tolerance for transient errors
  • Release payloads stored in two directories:
    • manifests/: CVO's own manifests
    • release-manifests/: Manifests for other cluster operators
  • Task graph orchestrates ordered application of manifests based on dependencies

Resource Libraries (lib/)

  • resourcebuilder/: Builds Kubernetes resources from manifests
  • resourceapply/: Applies resources with proper merge semantics for different API types
  • resourcemerge/: Merges resource specifications
  • resourceread/: Reads and validates manifests
  • capability/: Manages cluster capability filtering

Note: lib/resourcebuilder/resourcebuilder.go and lib/resourceread/resourceread.go are auto-generated by hack/generate-lib-resources.py.

Update Service Integration (pkg/cincinnati/)

  • Cincinnati client that communicates with OSUS to fetch available updates
  • Cincinnati is an update protocol for representing transitions between releases and facilitating automatic updates
  • Fetches and parses update graphs with nodes (release versions) and edges (upgrade paths)
  • Supports conditional edges based on cluster-specific conditions
  • Returns both unconditional and conditional update recommendations

Preconditions (pkg/payload/precondition/)

  • Validates cluster readiness before applying updates
  • Checks prevent upgrades when cluster is in unhealthy state
  • ClusterVersion-specific preconditions in precondition/clusterversion/

Entry Points

cmd/cluster-version-operator/main.go

  • Uses cobra for CLI structure
  • Actual start logic delegated to pkg/start/ package
pkg/start/start.go
  • Initializes and launches core CVO control loops
  • Sets up client connections, informers, and signal handling
  • Creates and starts the main Operator instance

cmd/cluster-version-operator-tests/main.go

  • OpenShift tests extension for CVO integration tests
  • Uses openshift-tests-extension framework to contribute CVO tests to OpenShift test suites

Key Patterns

Release Payload Image Structure

  • Release payload images are OCI container images containing:
    • CVO binary at a specific version
    • Manifest files in manifests/ and release-manifests/
    • release-metadata file with Cincinnati update graph info
    • image-references file mapping component names to images

Update Flow

  1. CVO fetches available updates from configured upstream (OSUS)
  2. Updates stored in status.availableUpdates and status.conditionalUpdates of ClusterVersion resource
  3. User/admin sets spec.desiredUpdate to trigger upgrade
  4. Validates payload signatures
  5. CVO retrieves new release payload image
  6. Applies manifests systematically using task graph
  7. Monitors cluster operator readiness during rollout
  8. Updates ClusterVersion status conditions throughout process

Manifest Application Order

  • Task graph in pkg/payload/task_graph.go determines ordering
  • Dependencies between operators enforced during updates
  • Parallel application during reconciliation for faster recovery

Debugging CVO in Cluster

# Check CVO deployment and pods
oc get deployment -n openshift-cluster-version cluster-version-operator
oc get pods -n openshift-cluster-version -l k8s-app=cluster-version-operator

# View logs
oc logs -n openshift-cluster-version -l k8s-app=cluster-version-operator

# Inspect ClusterVersion status
oc get clusterversion version -o yaml
oc adm upgrade

Common File Locations

  • Manifests: install/ - YAML files for CVO deployment itself
  • Build scripts: hack/ - Shell scripts for building, testing, pushing
  • Integration tests: test/ - Separate integration test suite
  • Development documentation: docs/dev/ - Development guides and workflows
    • README.md - Building, testing, and publishing development CVO images
    • feed-cvo-custom-graphs.md - Testing with custom update graphs
    • run-cvo-locally.md - Running CVO outside cluster

Commit Message Format

Follow the conventional format with subsystem prefix:

<subsystem>: <what changed>

<why this change was made>

Fixes #<issue-number>

Subsystems include: pkg/cvo, pkg/payload, lib/resourceapply, hack, etc.

Important Notes

ClusterVersion Resource

  • The ClusterVersion resource is the primary interface for configuring and monitoring CVO
  • Follows Kubernetes conventions: spec defines desired state, status reflects observed state
  • Users interact with CVO by setting spec.desiredUpdate to trigger upgrades
  • Status fields like availableUpdates and conditionalUpdates show available upgrade paths

Deployment and Operation

  • CVO runs as a single replica in the openshift-cluster-version namespace when deployed using repository manifests
  • Uses leader election to ensure only one active instance
  • Release payload images must be cryptographically verified before application (signature checking)
  • Capabilities system filters which manifests are applied based on the cluster's capability set

Development and Testing

  • Never test against production clusters - always use disposable test environments
  • CVO has significant control over cluster state and can disrupt operations during development