test(functional): add k8sfunctional and functional suites with kind CI - #1283
Open
TerryHowe wants to merge 10 commits into
Open
test(functional): add k8sfunctional and functional suites with kind CI#1283TerryHowe wants to merge 10 commits into
TerryHowe wants to merge 10 commits into
Conversation
TerryHowe
requested review from
Wwwsylvia,
sabre1041 and
shizhMSFT
as code owners
August 2, 2026 13:06
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1283 +/- ##
==========================================
+ Coverage 85.17% 85.20% +0.02%
==========================================
Files 94 94
Lines 7333 7333
==========================================
+ Hits 6246 6248 +2
+ Misses 664 662 -2
Partials 423 423 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
TerryHowe
force-pushed
the
test/functional-suite
branch
from
August 2, 2026 13:12
0c26f21 to
bdf128c
Compare
TerryHowe
force-pushed
the
test/functional-suite
branch
from
August 7, 2026 13:29
bdf128c to
7e973c6
Compare
TerryHowe
added a commit
to TerryHowe/oras-go
that referenced
this pull request
Aug 7, 2026
…ignature) Adds the second half of the functional module, behind the 'functional' build tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then exercises config-driven construction, mirror fallback, policy enforcement, and simple-signing verification end-to-end. BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and ClientBuilder packages, which are not yet on main, so the module does not build or 'go mod tidy' against main today. go.mod/go.sum here carry the full dependency set for the unblocked end state. A companion CI job runs the suite via setup.sh once it is unblocked. Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything (oras-project#1130) v3 breakdown. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TerryHowe
force-pushed
the
test/functional-suite
branch
from
August 11, 2026 17:29
7e973c6 to
9de5a3d
Compare
TerryHowe
added a commit
to TerryHowe/oras-go
that referenced
this pull request
Aug 11, 2026
…ignature) Adds the second half of the functional module, behind the 'functional' build tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then exercises config-driven construction, mirror fallback, policy enforcement, and simple-signing verification end-to-end. BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and ClientBuilder packages, which are not yet on main, so the module does not build or 'go mod tidy' against main today. go.mod/go.sum here carry the full dependency set for the unblocked end state. A companion CI job runs the suite via setup.sh once it is unblocked. Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything (oras-project#1130) v3 breakdown. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TerryHowe
force-pushed
the
test/functional-suite
branch
from
August 24, 2026 01:27
971b7c2 to
ca3f27d
Compare
TerryHowe
added a commit
to TerryHowe/oras-go
that referenced
this pull request
Aug 24, 2026
…ignature) Adds the second half of the functional module, behind the 'functional' build tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then exercises config-driven construction, mirror fallback, policy enforcement, and simple-signing verification end-to-end. BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and ClientBuilder packages, which are not yet on main, so the module does not build or 'go mod tidy' against main today. go.mod/go.sum here carry the full dependency set for the unblocked end state. A companion CI job runs the suite via setup.sh once it is unblocked. Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything (oras-project#1130) v3 breakdown. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TerryHowe
force-pushed
the
test/functional-suite
branch
from
August 24, 2026 14:43
ca3f27d to
dd9d9a5
Compare
TerryHowe
added a commit
to TerryHowe/oras-go
that referenced
this pull request
Aug 24, 2026
The functional tests are reviewed on their own in oras-project#1283 and are not required for the v3 release, so carrying a second copy here only invites the two to drift. With them and the objects package gone, this branch is the set of changes v3 actually needs. Removes test/functional entirely, along with its wiring: the functional-test job in build.yml, the test-functional target in the Makefile, and the .gitignore entry for its run state. Makefile and .gitignore are now identical to main again. Kept the "permissions: contents: read" block added to build.yml. It hardens the build workflow generally and has nothing to do with the functional suites. The root module builds, vets and tests clean, and is now the only Go module in the tree. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TerryHowe
marked this pull request as draft
August 24, 2026 21:54
Adds a standalone functional integration module under test/functional (its own go.mod, wired to the library via replace => ../../). The suite is behind the k8sfunctional build tag: TestMain stands up zot via Kubernetes manifests (testdata/k8s), or targets a pre-existing registry with ORAS_FUNCTIONAL_SKIP_DEPLOY, and exercises the public API end-to-end — push/pull, tags, referrers, copy, delete, registry catalog, and auth (with a credential helper built on the fly). This is the portion of the larger functional suite that compiles and runs against the current main API. The complementary 'functional'-tagged suite (config-, mirror-, and signature-driven tests) depends on packages still landing via the v3 breakdown and arrives in a stacked follow-up. Part of the feat/everything (oras-project#1130) v3 breakdown. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Adds .github/workflows/functional.yml. The job provisions an ephemeral Kubernetes cluster with kind and runs the test/functional module with the k8sfunctional build tag; TestMain deploys zot from testdata/k8s and the tests exercise the public API end-to-end. Runs on pushes and PRs to main that touch Go sources or the suite. Actions are pinned to commit SHAs per repo convention. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
…ignature) Adds the second half of the functional module, behind the 'functional' build tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then exercises config-driven construction, mirror fallback, policy enforcement, and simple-signing verification end-to-end. BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and ClientBuilder packages, which are not yet on main, so the module does not build or 'go mod tidy' against main today. go.mod/go.sum here carry the full dependency set for the unblocked end state. A companion CI job runs the suite via setup.sh once it is unblocked. Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything (oras-project#1130) v3 breakdown. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
The test/functional module still declared go 1.24.0 and x/sync v0.19.0 while the replaced parent module moved to go 1.25.0 and x/sync v0.22.0, failing both functional jobs with "updates to go.mod needed". Also update helpers_test.go for oras-project#1297, which hoisted Client and PlainHTTP from Repository to Registry. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
…ature TestSignature_GPG_WrongKeyRejected signs with key A and verifies against key B, then asserted IsImageAllowed returned a nil error. That held until oras-project#1316, which deliberately changed the contract: Verify now collects the per-signature rejection reasons and returns them, so a misconfigured keyring is distinguishable from genuinely unsigned content. Only the genuinely unsigned case still returns (false, nil). Assert the current contract instead: the image must be rejected, and the rejection must come with a reason. The unsigned half of the contract is already covered by TestSignature_GPG_UnsignedImageRejected, which expects a nil error and passes. Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TerryHowe
force-pushed
the
test/functional-suite
branch
from
September 4, 2026 17:32
dd9d9a5 to
728b33b
Compare
TerryHowe
marked this pull request as ready for review
September 4, 2026 17:40
The functional job set FUNCTIONAL_TEST_MIRROR while the suite reads FUNCTIONAL_TEST_MIRROR_REGISTRY, never set FUNCTIONAL_TEST_CERTS_DIR at all, and never sourced the env file setup.sh writes both of them to. Every mirror test therefore hit its "not configured" skip and the job went green having exercised no mirror code. mirror_test.go is the only caller of NewClientBuilder and NewRepositoryWithProperties, so config-driven repository construction had no effective coverage either. Source /tmp/oras-functional-env after setup.sh rather than restating the endpoints in the workflow, so a port change in setup.sh cannot silently desynchronise the two again. Add FUNCTIONAL_TEST_REQUIRE_MIRROR, set by the job, which turns the skip into a failure. Skipping is right for an ad-hoc local run without the full setup; in CI the mirror is supposed to be there, and a skip is a defect to be reported rather than a state to pass through. The certs.d plain-HTTP fallback becomes fatal under the same flag, since setup.sh serves the mirror over TLS and the fallback would otherwise surface as an opaque protocol error. Also drop the BLOCKED comment: config (oras-project#1225), signature (oras-project#1279) and ClientBuilder (oras-project#1328) have all landed, and the suite builds against main. Signed-off-by: Terry Howe <terrylhowe@gmail.com> Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33 Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Every negative assertion in the suite checked only that some error came back.
TestUnauthenticatedPushFails passed just as happily on a connection refused as
on a 401, and the mirror bypass tests passed whether the primary returned a
404 or the mirror failed its TLS handshake -- which is precisely the failure
they exist to distinguish.
Pin the error identity instead:
- unauthenticated push -> auth.ErrBasicCredentialNotFound, the sentinel for
a Basic challenge with no credential configured
- wrong credentials -> an *errcode.ErrorResponse carrying 401, which proves
the request reached the registry rather than failing before authenticating
- resolve after delete, and both mirror-bypass tests -> errdef.ErrNotFound
The policy denials have no sentinel to match -- the policy layer formats them
with fmt.Errorf -- so requirePolicyDenied still matches on the message, but
now on the full text including the reference the check ran against. Which
reference a policy check is computed for is where scope handling goes wrong,
and the previous "access denied" substring would not have noticed.
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Neither suite touched manifest lists: no index was ever pushed, resolved, fetched or copied, and CopyOptions.WithTargetPlatform and ResolveOptions.TargetPlatform had no functional coverage at all. Add a two-platform index to each suite -- index_test.go against zot under the k8sfunctional tag, multiarch_test.go against distribution under the functional tag -- so index handling is exercised on both registry implementations the suites already run. Covered: push and resolve an index by tag and get the index media type back; read it back and confirm the registry preserved the entries and their platforms; walk it with content.Successors; copy the whole graph and verify both children and every blob they reference arrived. For platform selection, the copy root must become the matching child manifest rather than the index, the sibling platform must be left behind, and a platform absent from the index must fail with errdef.ErrNotFound rather than falling back to the index or to an arbitrary child. The helpers are duplicated across the two files because the suites have separate helper sets; a shared test package would absorb both. Signed-off-by: Terry Howe <terrylhowe@gmail.com> Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33 Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Twelve files were not gofmt-clean, most of them because the //go:build line sat below the license header instead of above it. Both toolchains the repo targets (1.25 and 1.26) canonicalise it to the top of the file, and some of the suite's own files were already written that way, so the tree was inconsistent with itself as well as with gofmt. The remainder is an import reorder and struct-literal realignment in infra/registry.go. Formatting only: both tags vet, compile, and still gate correctly -- an untagged 'go test ./...' selects no tests from either suite. Signed-off-by: Terry Howe <terrylhowe@gmail.com> Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33 Signed-off-by: Terry Howe <terrylhowe@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The functional test suites for v3, extracted from #1130. Combines what were
previously two PRs — #1284 has been folded in here and closed, since it was a
strict superset of this branch and splitting them only meant rebasing the same
files twice.
Two suites, each behind its own build tag so neither runs during
go test ./...:k8sfunctional— the integration suite, run against a registry deployedon a kind cluster in CI.
functional— config, mirror, policy and signature coverage, run againstregistries brought up by
setup.sh.The tags are mutually exclusive: each suite defines its own
TestMain, sogo vet -tags 'functional k8sfunctional'reports multiple definitions. Vet ortest one tag at a time.
Base
Rebased onto
main. #1328 has merged, so the twoClientBuildercommits thatused to sit at the base of this branch have dropped out — the diff is now
test/functional/and.github/workflows/functional.ymlonly, and both suitesbuild against
mainas it stands (go vetandgo test -cclean under eachtag).
One behavioural fix worth a look
TestSignature_GPG_WrongKeyRejectedsigns with key A and verifies againstkey B, and asserted that
IsImageAllowedreturned a nil error. That wascorrect until #1316, which deliberately changed the contract:
So a signature that is present but validates against no configured key now
rejects with a reason, while genuinely unsigned content rejects with a nil
error. The test asserted the old half of that contract and has been red since
#1316 merged. It now asserts the current one: the image must be rejected, and
the rejection must carry a reason. The unsigned half stays covered by
TestSignature_GPG_UnsignedImageRejected, which expects a nil error.Both paths deny the image, so this was never a policy bypass — only a stale
expectation in the test.