Skip to content

Commit 0251152

Browse files
HiranAdikariclaude
andcommitted
Route image create and resolve through the agent
The image slice of the credential-locality work: CreateImage and resolveImage now go through the cluster-access seam, so image creation and lookup work in a remote (agent-only) zone. - vmImageCapability gains VerbCreate (RouteVerbs) and Create/Apply (AgentVerbs); the agent RBAC is regenerated to grant create + patch on virtualmachineimages (patch because the agent create is server-side apply). - resolveImage lists via c.access.List (was a direct c.dynamic list), so image resolution routes to the zone's agent. - CreateImage routes via c.access.Create. Because the agent create is SSA and SSA requires a name, the image now gets a client-assigned "image-<random>" name instead of the apiserver's generateName; the local path still POSTs it and Harvester imports it the same way. resolveImage matches by display name / ID, so the assigned name is not user-visible. Both modules build; unit tests (new create/resolve seam tests and the RBAC drift check) pass. CreateImage builds a VirtualMachineImage CR, so a live structural diff of the generated object against a real Harvester image is the remaining pre-merge check (no cluster here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC3BDHCFmJBdxvGVW69FAr
1 parent a6452cf commit 0251152

6 files changed

Lines changed: 426 additions & 60 deletions

File tree

dc-api/internal/providers/clusteraccess/capabilities.go

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,10 @@ var vmCapability = AgentCapability{
7575

7676
// The network families (NAD, Vpc, Subnet) are onboarded for the kubeovn CRD
7777
// CRUD slice (RouteVerbs/AgentVerbs filled below). vmImageCapability is onboarded
78-
// for the read/list path (ListImages routed via the agent — RouteVerbs={Get,List},
79-
// AgentVerbs={get,list}). vmiCapability remains a GVK-only pre-seeded entry:
78+
// for the read/list path AND image create (ListImages/resolveImage/CreateImage
79+
// routed via the agent — RouteVerbs={Get,List,Create}, AgentVerbs={get,list,
80+
// create,patch}; create+patch because SSA is the agent's create mechanism).
81+
// vmiCapability remains a GVK-only pre-seeded entry:
8082
// DefaultGVKMapper still resolves its GVR (the table stays a superset the drivers
8183
// rely on) but RouteVerbs=nil means it contributes nothing to the routing
8284
// allow-set and AgentVerbs=nil means it emits no RBAC rule. Onboarding it is a
@@ -93,14 +95,17 @@ var (
9395
APIVersion: "harvesterhci.io/v1beta1",
9496
Kind: "VirtualMachineImage",
9597
Namespaced: true,
96-
// Onboarded for the read path only: ListImages routes the cross-namespace
97-
// image catalog read through the agent (VerbList), and VerbGet rounds out
98-
// the read family. No write verbs — image create/import stays local-only.
99-
RouteVerbs: []Verb{VerbGet, VerbList},
100-
// SA grant: get/list on virtualmachineimages so the agent can serve the
101-
// image catalog list (and a future per-image get). No watch — there is no
102-
// routed watch path, so granting it would be unused RBAC (least-privilege).
103-
AgentVerbs: []Verb{VerbGet, VerbList},
98+
// Onboarded for the read/list path AND image create: ListImages routes the
99+
// cross-namespace catalog read (VerbList) and resolveImage the create-time
100+
// storageClass lookup, VerbGet rounds out the read family, and CreateImage
101+
// routes the image import (VerbCreate). No delete — image deletion stays
102+
// local-only for now.
103+
RouteVerbs: []Verb{VerbGet, VerbList, VerbCreate},
104+
// SA grant: get/list to serve the catalog read + resolveImage lookup, plus
105+
// create+patch because the AgentBacked create is a server-side apply
106+
// (VerbApply→patch). Mirrors vmCapability/nadCapability's write grant. No
107+
// watch — there is no routed watch path, so granting it would be unused RBAC.
108+
AgentVerbs: []Verb{VerbGet, VerbList, VerbCreate, VerbApply},
104109
}
105110
// nadCapability onboards the NetworkAttachmentDefinition CRUD that
106111
// CreateSubnet/DeleteSubnet route through the kubeovn seam.
@@ -142,7 +147,7 @@ var (
142147
//
143148
// Onboarded (RouteVerbs + AgentVerbs set): vmCapability, the three network
144149
// families nadCapability/vpcCapability/subnetCapability (the kubeovn CRD CRUD
145-
// slice), and vmImageCapability (read/list path). vmiCapability stays a GVK-only
150+
// slice), and vmImageCapability (read/list path + image create). vmiCapability stays a GVK-only
146151
// pre-seeded entry that keeps the wire mapper a superset without granting any
147152
// routing or RBAC. New families are added here (one struct each) in later
148153
// phases — never by editing the mapper, the allow-set switch, or the RBAC YAML
@@ -190,7 +195,9 @@ func buildDerived() {
190195
// set of seam Verbs that MAY route to the agent for ANY family. With the VM,
191196
// network, and image families onboarded this equals
192197
// {VerbGet, VerbList, VerbCreate, VerbDelete} (List added when ListVMs/Images/
193-
// Networks were routed through the agent). agentDecision consults this for
198+
// Networks were routed through the agent; the image family also routes VerbCreate
199+
// for CreateImage, but VM/NAD already contribute it so the union is unchanged).
200+
// agentDecision consults this for
194201
// membership, then gates reads vs writes by the per-family env toggles
195202
// (VerbList falls on the read side — see IsReadVerb).
196203
//

dc-api/internal/providers/clusteraccess/capabilities_test.go

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,7 @@ func TestRoutableVerbs_UnknownGVRNotRoutable(t *testing.T) {
7575

7676
// TestRoutableVerbs_ListRoutedForListFamilies asserts VerbList is routable for
7777
// exactly the families whose harvester list ops now route through the agent: the
78-
// VM, NAD, and image families. The image family routes List (and Get) but NOT
79-
// any write verb.
78+
// VM, NAD, and image families.
8079
func TestRoutableVerbs_ListRoutedForListFamilies(t *testing.T) {
8180
listFamilies := []schema.GroupVersionResource{
8281
{Group: "kubevirt.io", Version: "v1", Resource: "virtualmachines"},
@@ -93,17 +92,31 @@ func TestRoutableVerbs_ListRoutedForListFamilies(t *testing.T) {
9392
t.Errorf("%s must route VerbList", gvr.Resource)
9493
}
9594
}
95+
}
9696

97-
// The image family is read/list only — it must NOT route any write verb.
97+
// TestRoutableVerbs_ImageRoutesGetListCreate pins the image family's routable set
98+
// to exactly {Get, List, Create}: the read/list path (resolveImage/ListImages)
99+
// plus CreateImage's create (the image slice). Delete/Apply/Update must NOT be
100+
// routable — image deletion and generic apply stay local-only.
101+
func TestRoutableVerbs_ImageRoutesGetListCreate(t *testing.T) {
98102
imgGVR := schema.GroupVersionResource{Group: "harvesterhci.io", Version: "v1beta1", Resource: "virtualmachineimages"}
99-
imgVerbs, _ := RoutableVerbs(imgGVR)
100-
for _, w := range []Verb{VerbCreate, VerbApply, VerbUpdate, VerbDelete} {
103+
imgVerbs, ok := RoutableVerbs(imgGVR)
104+
if !ok {
105+
t.Fatal("virtualmachineimages must be an onboarded routable family")
106+
}
107+
for _, want := range []Verb{VerbGet, VerbList, VerbCreate} {
108+
if !imgVerbs[want] {
109+
t.Errorf("virtualmachineimages must route %v, got %v", want, imgVerbs)
110+
}
111+
}
112+
// CreateImage routes VerbCreate but NOT the other write verbs.
113+
for _, w := range []Verb{VerbApply, VerbUpdate, VerbDelete} {
101114
if imgVerbs[w] {
102-
t.Errorf("virtualmachineimages must NOT route write verb %v", w)
115+
t.Errorf("virtualmachineimages must NOT route %v", w)
103116
}
104117
}
105-
if !imgVerbs[VerbGet] || !imgVerbs[VerbList] {
106-
t.Errorf("virtualmachineimages must route Get+List, got %v", imgVerbs)
118+
if len(imgVerbs) != 3 {
119+
t.Errorf("virtualmachineimages routable set = %v, want exactly {Get, List, Create}", imgVerbs)
107120
}
108121
}
109122

dc-api/internal/providers/harvester/client.go

Lines changed: 72 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,10 @@ package harvester
2626

2727
import (
2828
"context"
29+
"crypto/rand"
2930
"crypto/sha256"
3031
"encoding/base64"
32+
"encoding/hex"
3133
"encoding/json"
3234
"fmt"
3335
"strings"
@@ -97,40 +99,43 @@ type Client struct {
9799
// clusteraccess.Direct wrapping `dynamic` (today's behaviour, byte-identical)
98100
// unless WithRoutedAccessor injects a routed accessor. The VM-object ops route
99101
// through it: GetVM's primary VirtualMachine read (read slice), CreateVM's
100-
// create + DeleteVM's delete (write slice 1), and the collection reads
101-
// ListVMs/ListImages/ListNetworks (list slice — M-D). Image RESOLUTION (the
102-
// create-time storageClass lookup), the cloud-provider SA bootstrap, and the
103-
// VMI read inside GetVM still call c.dynamic directly. The seam never leaks an
104-
// agent concept past the ComputeProvider interface.
102+
// create + DeleteVM's delete (write slice 1), the collection reads
103+
// ListVMs/ListImages/ListNetworks (list slice — M-D), and the image slice —
104+
// CreateImage's create and resolveImage's create-time storageClass lookup. The
105+
// cloud-provider SA bootstrap and the VMI read inside GetVM still call
106+
// c.dynamic directly. The seam never leaks an agent concept past the
107+
// ComputeProvider interface.
105108
access clusteraccess.Accessor
106109

107110
// remoteRegion/remoteZone are non-empty ONLY for a credential-free REMOTE
108111
// client built by NewRemoteClient (multi-zone routing). For such a client
109-
// c.dynamic is nil: the seam ops (CreateVM/GetVM/DeleteVM via c.access, and
110-
// the list reads via c.access.List) route to that zone's agent. The two
112+
// c.dynamic is nil: the seam ops (CreateVM/GetVM/DeleteVM via c.access, the
113+
// list reads via c.access.List, and the image slice — CreateImage's create and
114+
// resolveImage's lookup — via c.access) route to that zone's agent. The two
111115
// remaining direct paths behave differently on a remote client:
112116
// - GetVM's VMI IP enrichment read DEGRADES GRACEFULLY: it is SKIPPED (the
113117
// `c.dynamic == nil` guard returns the agent-supplied VM status WITHOUT IP
114118
// enrichment — no error), so a remote VM's status is still reported.
115-
// - the genuinely local-only ops — image RESOLUTION/import, the cloud-provider
116-
// SA bootstrap, and VM create's local storageClass resolution — have no
117-
// direct path and return a clear local-only error (localOnlyErr) naming the
118-
// zone instead of panicking on a nil dynamic client.
119+
// - the genuinely local-only ops — the cloud-provider SA bootstrap and VM
120+
// create's local storageClass resolution — have no direct path and return a
121+
// clear local-only error (localOnlyErr) naming the zone instead of panicking
122+
// on a nil dynamic client.
119123
// Empty for the LOCAL client → today's behaviour.
120124
remoteRegion, remoteZone string
121125
}
122126

123-
// localOnlyErr is returned by a REMOTE client's direct-only methods (image
124-
// resolution/import, the cloud-provider SA bootstrap, VM create which needs the
125-
// image storageClass resolved locally). These touch c.dynamic, which a remote
126-
// client does not have. Failing here — BEFORE a PENDING row or a provisioner
127-
// call — is the documented local-only constraint for the remote-zone build:
128-
// routing the VM-object CRUD lifecycle and the collection reads is in scope;
129-
// routing image resolution/import and the SA bootstrap is future work behind this
130-
// explicit error.
127+
// localOnlyErr is returned by a REMOTE client's direct-only methods: VM create
128+
// (which resolves the image storageClass — see CreateVM's note) and the
129+
// cloud-provider SA bootstrap. These touch c.dynamic, which a remote client does
130+
// not have. Failing here — BEFORE a PENDING row or a provisioner call — is the
131+
// documented local-only constraint for the remote-zone build. NOTE: image
132+
// resolution/import (resolveImage, CreateImage) and the collection reads are NO
133+
// LONGER local-only — they route through the cluster-access seam (c.access), so a
134+
// remote client serves them via the agent; only VM create's storageClass lookup
135+
// and the SA bootstrap remain behind this explicit error.
131136
func (c *Client) localOnlyErr(op string) error {
132137
return fmt.Errorf(
133-
"%s is not supported for remote zone %s/%s yet: dc-api holds no direct Harvester credentials there (image resolution/import and the cloud-provider SA bootstrap are local-only for now)",
138+
"%s is not supported for remote zone %s/%s yet: dc-api holds no direct Harvester credentials there (VM create's storageClass resolution and the cloud-provider SA bootstrap are local-only for now)",
134139
op, c.remoteRegion, c.remoteZone)
135140
}
136141

@@ -225,10 +230,11 @@ func (c *Client) Name() string { return "harvester" }
225230
// is a handler-layer bug, not a recoverable provider condition.
226231
func (c *Client) CreateVM(ctx context.Context, tenantID, projectID string, spec models.VMSpec) (*models.Resource, error) {
227232
if c.dynamic == nil {
228-
// REMOTE client: VM create needs the image template resolved against the
229-
// zone's local VirtualMachineImage catalog (resolveImage → c.dynamic),
230-
// which we don't have for a remote zone. Fail clearly BEFORE building the
231-
// manifest so the handler surfaces a useful error rather than misrouting.
233+
// REMOTE client: routing the full VM create lifecycle to a remote zone is a
234+
// later slice (resolveImage now routes through c.access, but wiring the whole
235+
// create — manifest build, MAC pinning, DNS injection — for a remote zone is
236+
// out of scope here). Fail clearly BEFORE building the manifest so the handler
237+
// surfaces a useful error rather than misrouting.
232238
return nil, c.localOnlyErr("VM create")
233239
}
234240
ns := common.NamespaceForProject(tenantID, projectID)
@@ -753,20 +759,38 @@ func (c *Client) ListNetworks(ctx context.Context) ([]*models.Network, error) {
753759
// CreateImage creates a VirtualMachineImage CRD in Harvester, which triggers
754760
// Harvester to download the image from the given URL into Longhorn storage.
755761
// The image is available for VM creation once its status transitions to "active".
762+
//
763+
// The object is created through the cluster-access seam (c.access.Create): the
764+
// Direct path (toggle OFF) is a plain dynamic POST — byte-identical to the
765+
// pre-seam behaviour except that the name is now client-assigned; the agent path
766+
// (toggle ON + live agent) is a server-side apply, the agent's only create
767+
// mechanism. SSA REQUIRES a name (generateName cannot SSA), so we assign a fixed
768+
// "image-<rand>" name here rather than relying on the apiserver's generateName.
769+
// A POST accepts the explicit name fine, so both seams agree. A REMOTE client
770+
// (c.dynamic nil) now serves this through the agent — it no longer returns
771+
// localOnlyErr.
756772
func (c *Client) CreateImage(ctx context.Context, displayName, downloadURL string) (*models.Image, error) {
757-
if c.dynamic == nil {
758-
return nil, c.localOnlyErr("CreateImage")
759-
}
760773
// Images are created in the "default" namespace in Harvester.
761774
const imageNamespace = "default"
762775

776+
// Client-assigned name: SSA (the agent create path) has no generateName
777+
// equivalent, so we must supply metadata.name. The short crypto-random suffix
778+
// mirrors what the apiserver's generateName would have produced ("image-XXXXX")
779+
// and keeps names collision-resistant. resolveImage matches by full ID, name,
780+
// OR displayName, so a client-assigned name is transparent to VM create.
781+
suffix, err := randHexSuffix()
782+
if err != nil {
783+
return nil, fmt.Errorf("harvester create image %q: generate name suffix: %w", displayName, err)
784+
}
785+
name := "image-" + suffix
786+
763787
obj := &unstructured.Unstructured{
764788
Object: map[string]interface{}{
765789
"apiVersion": "harvesterhci.io/v1beta1",
766790
"kind": "VirtualMachineImage",
767791
"metadata": map[string]interface{}{
768-
"generateName": "image-",
769-
"namespace": imageNamespace,
792+
"name": name,
793+
"namespace": imageNamespace,
770794
"labels": map[string]interface{}{
771795
"dc-api/managed": "true",
772796
},
@@ -779,7 +803,7 @@ func (c *Client) CreateImage(ctx context.Context, displayName, downloadURL strin
779803
},
780804
}
781805

782-
created, err := c.dynamic.Resource(vmImageGVR).Namespace(imageNamespace).Create(ctx, obj, metav1.CreateOptions{})
806+
created, err := c.access.Create(ctx, vmImageGVR, imageNamespace, obj, metav1.CreateOptions{})
783807
if err != nil {
784808
return nil, fmt.Errorf("harvester create image %q: %w", displayName, err)
785809
}
@@ -791,6 +815,18 @@ func (c *Client) CreateImage(ctx context.Context, displayName, downloadURL strin
791815
}, nil
792816
}
793817

818+
// randHexSuffix returns 8 hex characters (4 crypto-random bytes) for a
819+
// client-assigned image name. It mirrors the "image-XXXXX" shape the apiserver's
820+
// generateName produced, but is chosen here because the agent create path is a
821+
// server-side apply, which requires a name up front.
822+
func randHexSuffix() (string, error) {
823+
b := make([]byte, 4)
824+
if _, err := rand.Read(b); err != nil {
825+
return "", err
826+
}
827+
return hex.EncodeToString(b), nil
828+
}
829+
794830
// resolveImage resolves a user-supplied image string to a "namespace/resource-name" ID
795831
// and the storage class Harvester created for it.
796832
//
@@ -802,7 +838,12 @@ func (c *Client) CreateImage(ctx context.Context, displayName, downloadURL strin
802838
// - A full ID: "default/image-abc123" (looked up by namespace+name)
803839
// - A display name: "ubuntu-22.04" (looked up by spec.displayName)
804840
func (c *Client) resolveImage(ctx context.Context, nameOrID string) (imageID, storageClass string, err error) {
805-
list, err := c.dynamic.Resource(vmImageGVR).Namespace("").List(ctx, metav1.ListOptions{})
841+
// Routed through the cluster-access seam (c.access): Direct (byte-identical
842+
// cross-namespace dynamic List) when the toggle is OFF, Session.List when ON
843+
// and an agent serves the zone. This is why a REMOTE client (c.dynamic nil)
844+
// can now resolve an image — the lookup no longer touches c.dynamic. Field
845+
// extraction (displayName, status.storageClassName) below is seam-agnostic.
846+
list, err := c.access.List(ctx, vmImageGVR, "", metav1.ListOptions{})
806847
if err != nil {
807848
return "", "", fmt.Errorf("list images for lookup: %w", err)
808849
}

0 commit comments

Comments
 (0)