Skip to content

test(functional): add k8sfunctional and functional suites with kind CI - #1283

Open
TerryHowe wants to merge 10 commits into
oras-project:mainfrom
TerryHowe:test/functional-suite
Open

test(functional): add k8sfunctional and functional suites with kind CI#1283
TerryHowe wants to merge 10 commits into
oras-project:mainfrom
TerryHowe:test/functional-suite

Conversation

@TerryHowe

@TerryHowe TerryHowe commented Aug 2, 2026

Copy link
Copy Markdown
Member

What

The functional test suites for v3, extracted from #1130. Combines what were
previously two PRs — #1284 has been folded in here and closed, since it was a
strict superset of this branch and splitting them only meant rebasing the same
files twice.

Two suites, each behind its own build tag so neither runs during go test ./...:

  • k8sfunctional — the integration suite, run against a registry deployed
    on a kind cluster in CI.
  • functional — config, mirror, policy and signature coverage, run against
    registries brought up by setup.sh.

The tags are mutually exclusive: each suite defines its own TestMain, so
go vet -tags 'functional k8sfunctional' reports multiple definitions. Vet or
test one tag at a time.

Base

Rebased onto main. #1328 has merged, so the two ClientBuilder commits that
used to sit at the base of this branch have dropped out — the diff is now
test/functional/ and .github/workflows/functional.yml only, and both suites
build against main as it stands (go vet and go test -c clean under each
tag).

One behavioural fix worth a look

TestSignature_GPG_WrongKeyRejected signs with key A and verifies against
key B, and asserted that IsImageAllowed returned a nil error. That was
correct until #1316, which deliberately changed the contract:

Verify discarded every per-signature rejection and returned (false, nil),
which made a misconfigured keyring indistinguishable from unsigned content.
Collect the reasons and return them when nothing validates. Genuinely
unsigned content still returns (false, nil).

So a signature that is present but validates against no configured key now
rejects with a reason, while genuinely unsigned content rejects with a nil
error. The test asserted the old half of that contract and has been red since
#1316 merged. It now asserts the current one: the image must be rejected, and
the rejection must carry a reason. The unsigned half stays covered by
TestSignature_GPG_UnsignedImageRejected, which expects a nil error.

Both paths deny the image, so this was never a policy bypass — only a stale
expectation in the test.

@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.20%. Comparing base (4329def) to head (4952800).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1283      +/-   ##
==========================================
+ Coverage   85.17%   85.20%   +0.02%     
==========================================
  Files          94       94              
  Lines        7333     7333              
==========================================
+ Hits         6246     6248       +2     
+ Misses        664      662       -2     
  Partials      423      423              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from 0c26f21 to bdf128c Compare August 2, 2026 13:12
@TerryHowe TerryHowe changed the title test(functional): add functional test suite + kind CI (main-compatible subset) test(functional): add k8sfunctional suite + kind CI (main-compatible) Aug 2, 2026
@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from bdf128c to 7e973c6 Compare August 7, 2026 13:29
TerryHowe added a commit to TerryHowe/oras-go that referenced this pull request Aug 7, 2026
…ignature)

Adds the second half of the functional module, behind the 'functional' build
tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via
setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then
exercises config-driven construction, mirror fallback, policy enforcement, and
simple-signing verification end-to-end.

BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and
ClientBuilder packages, which are not yet on main, so the module does not
build or 'go mod tidy' against main today. go.mod/go.sum here carry the full
dependency set for the unblocked end state. A companion CI job runs the suite
via setup.sh once it is unblocked.

Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything
(oras-project#1130) v3 breakdown.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from 7e973c6 to 9de5a3d Compare August 11, 2026 17:29
TerryHowe added a commit to TerryHowe/oras-go that referenced this pull request Aug 11, 2026
…ignature)

Adds the second half of the functional module, behind the 'functional' build
tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via
setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then
exercises config-driven construction, mirror fallback, policy enforcement, and
simple-signing verification end-to-end.

BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and
ClientBuilder packages, which are not yet on main, so the module does not
build or 'go mod tidy' against main today. go.mod/go.sum here carry the full
dependency set for the unblocked end state. A companion CI job runs the suite
via setup.sh once it is unblocked.

Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything
(oras-project#1130) v3 breakdown.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from 971b7c2 to ca3f27d Compare August 24, 2026 01:27
TerryHowe added a commit to TerryHowe/oras-go that referenced this pull request Aug 24, 2026
…ignature)

Adds the second half of the functional module, behind the 'functional' build
tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via
setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then
exercises config-driven construction, mirror fallback, policy enforcement, and
simple-signing verification end-to-end.

BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and
ClientBuilder packages, which are not yet on main, so the module does not
build or 'go mod tidy' against main today. go.mod/go.sum here carry the full
dependency set for the unblocked end state. A companion CI job runs the suite
via setup.sh once it is unblocked.

Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything
(oras-project#1130) v3 breakdown.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from ca3f27d to dd9d9a5 Compare August 24, 2026 14:43
@TerryHowe TerryHowe changed the title test(functional): add k8sfunctional suite + kind CI (main-compatible) test(functional): add k8sfunctional and functional suites with kind CI Aug 24, 2026
TerryHowe added a commit to TerryHowe/oras-go that referenced this pull request Aug 24, 2026
The functional tests are reviewed on their own in oras-project#1283 and are not required
for the v3 release, so carrying a second copy here only invites the two to
drift. With them and the objects package gone, this branch is the set of
changes v3 actually needs.

Removes test/functional entirely, along with its wiring: the functional-test
job in build.yml, the test-functional target in the Makefile, and the
.gitignore entry for its run state. Makefile and .gitignore are now identical
to main again.

Kept the "permissions: contents: read" block added to build.yml. It hardens
the build workflow generally and has nothing to do with the functional suites.

The root module builds, vets and tests clean, and is now the only Go module in
the tree.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
@TerryHowe
TerryHowe marked this pull request as draft August 24, 2026 21:54
Adds a standalone functional integration module under test/functional (its
own go.mod, wired to the library via replace => ../../). The suite is behind
the k8sfunctional build tag: TestMain stands up zot via Kubernetes manifests
(testdata/k8s), or targets a pre-existing registry with
ORAS_FUNCTIONAL_SKIP_DEPLOY, and exercises the public API end-to-end —
push/pull, tags, referrers, copy, delete, registry catalog, and auth (with a
credential helper built on the fly).

This is the portion of the larger functional suite that compiles and runs
against the current main API. The complementary 'functional'-tagged suite
(config-, mirror-, and signature-driven tests) depends on packages still
landing via the v3 breakdown and arrives in a stacked follow-up.

Part of the feat/everything (oras-project#1130) v3 breakdown.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Adds .github/workflows/functional.yml. The job provisions an ephemeral
Kubernetes cluster with kind and runs the test/functional module with the
k8sfunctional build tag; TestMain deploys zot from testdata/k8s and the tests
exercise the public API end-to-end. Runs on pushes and PRs to main that touch
Go sources or the suite. Actions are pinned to commit SHAs per repo convention.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
…ignature)

Adds the second half of the functional module, behind the 'functional' build
tag. Its TestMain (functional_test.go) deploys a primary + mirror registry via
setup.sh and reads endpoints from FUNCTIONAL_TEST_REGISTRY / _MIRROR, then
exercises config-driven construction, mirror fallback, policy enforcement, and
simple-signing verification end-to-end.

BLOCKED: these tests depend on the config (oras-project#1225), signature (oras-project#1279), and
ClientBuilder packages, which are not yet on main, so the module does not
build or 'go mod tidy' against main today. go.mod/go.sum here carry the full
dependency set for the unblocked end state. A companion CI job runs the suite
via setup.sh once it is unblocked.

Stacked on the k8sfunctional suite (oras-project#1283). Part of the feat/everything
(oras-project#1130) v3 breakdown.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
The test/functional module still declared go 1.24.0 and x/sync v0.19.0
while the replaced parent module moved to go 1.25.0 and x/sync v0.22.0,
failing both functional jobs with "updates to go.mod needed".

Also update helpers_test.go for oras-project#1297, which hoisted Client and
PlainHTTP from Repository to Registry.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
…ature

TestSignature_GPG_WrongKeyRejected signs with key A and verifies against
key B, then asserted IsImageAllowed returned a nil error. That held until
oras-project#1316, which deliberately changed the contract: Verify now collects the
per-signature rejection reasons and returns them, so a misconfigured keyring
is distinguishable from genuinely unsigned content. Only the genuinely
unsigned case still returns (false, nil).

Assert the current contract instead: the image must be rejected, and the
rejection must come with a reason. The unsigned half of the contract is
already covered by TestSignature_GPG_UnsignedImageRejected, which expects a
nil error and passes.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
@TerryHowe
TerryHowe force-pushed the test/functional-suite branch from dd9d9a5 to 728b33b Compare September 4, 2026 17:32
@TerryHowe
TerryHowe marked this pull request as ready for review September 4, 2026 17:40
The functional job set FUNCTIONAL_TEST_MIRROR while the suite reads
FUNCTIONAL_TEST_MIRROR_REGISTRY, never set FUNCTIONAL_TEST_CERTS_DIR at all,
and never sourced the env file setup.sh writes both of them to. Every mirror
test therefore hit its "not configured" skip and the job went green having
exercised no mirror code. mirror_test.go is the only caller of
NewClientBuilder and NewRepositoryWithProperties, so config-driven repository
construction had no effective coverage either.

Source /tmp/oras-functional-env after setup.sh rather than restating the
endpoints in the workflow, so a port change in setup.sh cannot silently
desynchronise the two again.

Add FUNCTIONAL_TEST_REQUIRE_MIRROR, set by the job, which turns the skip into
a failure. Skipping is right for an ad-hoc local run without the full setup;
in CI the mirror is supposed to be there, and a skip is a defect to be
reported rather than a state to pass through. The certs.d plain-HTTP fallback
becomes fatal under the same flag, since setup.sh serves the mirror over TLS
and the fallback would otherwise surface as an opaque protocol error.

Also drop the BLOCKED comment: config (oras-project#1225), signature (oras-project#1279) and
ClientBuilder (oras-project#1328) have all landed, and the suite builds against main.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Every negative assertion in the suite checked only that some error came back.
TestUnauthenticatedPushFails passed just as happily on a connection refused as
on a 401, and the mirror bypass tests passed whether the primary returned a
404 or the mirror failed its TLS handshake -- which is precisely the failure
they exist to distinguish.

Pin the error identity instead:

  - unauthenticated push -> auth.ErrBasicCredentialNotFound, the sentinel for
    a Basic challenge with no credential configured
  - wrong credentials -> an *errcode.ErrorResponse carrying 401, which proves
    the request reached the registry rather than failing before authenticating
  - resolve after delete, and both mirror-bypass tests -> errdef.ErrNotFound

The policy denials have no sentinel to match -- the policy layer formats them
with fmt.Errorf -- so requirePolicyDenied still matches on the message, but
now on the full text including the reference the check ran against. Which
reference a policy check is computed for is where scope handling goes wrong,
and the previous "access denied" substring would not have noticed.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Neither suite touched manifest lists: no index was ever pushed, resolved,
fetched or copied, and CopyOptions.WithTargetPlatform and
ResolveOptions.TargetPlatform had no functional coverage at all.

Add a two-platform index to each suite -- index_test.go against zot under the
k8sfunctional tag, multiarch_test.go against distribution under the functional
tag -- so index handling is exercised on both registry implementations the
suites already run.

Covered: push and resolve an index by tag and get the index media type back;
read it back and confirm the registry preserved the entries and their
platforms; walk it with content.Successors; copy the whole graph and verify
both children and every blob they reference arrived. For platform selection,
the copy root must become the matching child manifest rather than the index,
the sibling platform must be left behind, and a platform absent from the index
must fail with errdef.ErrNotFound rather than falling back to the index or to
an arbitrary child.

The helpers are duplicated across the two files because the suites have
separate helper sets; a shared test package would absorb both.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Twelve files were not gofmt-clean, most of them because the //go:build line sat
below the license header instead of above it. Both toolchains the repo targets
(1.25 and 1.26) canonicalise it to the top of the file, and some of the suite's
own files were already written that way, so the tree was inconsistent with
itself as well as with gofmt.

The remainder is an import reorder and struct-literal realignment in
infra/registry.go.

Formatting only: both tags vet, compile, and still gate correctly -- an
untagged 'go test ./...' selects no tests from either suite.

Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Claude-Session: https://claude.ai/code/session_01MpTKRChww8pvfSPgN7cp33
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant