Skip to content

CMP-3879: Create an e2e suite for tailoring tests WIP#1105

Draft
Anna-Koudelkova wants to merge 1 commit intoComplianceAsCode:masterfrom
Anna-Koudelkova:CMP-3879
Draft

CMP-3879: Create an e2e suite for tailoring tests WIP#1105
Anna-Koudelkova wants to merge 1 commit intoComplianceAsCode:masterfrom
Anna-Koudelkova:CMP-3879

Conversation

@Anna-Koudelkova
Copy link
Copy Markdown
Collaborator

@Anna-Koudelkova Anna-Koudelkova commented Mar 11, 2026

First attempt to start with the refactor based on feature groups. Not complete as we have not finished porting all the test cases upstream and also I need some guidance on importance of the test cases.

This is heavily coming from the refactor doc, trying to cover the tailoring tests.

  1. Test case execution order
    One thing I have found out is that golang would go through all the test cases and when it is parallel, it puts it on hold and run the serial test cases one by one first and the parallel test cases gets to run last.

  2. Flags
    For the test purposes I have labeled 4 test cases as Critical based on Cursor suggestion - it does not reflect the real importance!
    I was unable to use the Critical flag simply as part of the name of the test case (to run all the critical test cases only), because some of the CRDs like ScanSettings etc. gets named after the name of the tests and it cannot handle the underscores ( I was using TestName_Critical at first) and fails.
    That is why I resulted into filtering the test cases using flag.Bool() which seems to be working now:

make e2e-tailoring-critical   E2E_CONTENT_IMAGE_PATH="quay.io/rh-ee-akoudelk/content_repo:testcontent"  E2E_BROKEN_CONTENT_IMAGE_PATH="ghcr.io/complianceascode/test-broken-content-ocp"  E2E_GO_TEST_FLAGS="-v -timeout 60m"
..
=== RUN   TestScanTailoredProfileIsDeprecated
=== PAUSE TestScanTailoredProfileIsDeprecated
=== RUN   TestScanTailoredProfileHasDuplicateVariables
    main_test.go:128: Skipping non-critical test
--- SKIP: TestScanTailoredProfileHasDuplicateVariables (0.00s)
=== RUN   TestSingleTailoredScanSucceeds
=== PAUSE TestSingleTailoredScanSucceeds
=== RUN   TestScanSettingBindingTailoringManyEnablingRulePass
    main_test.go:293: Skipping non-critical test
--- SKIP: TestScanSettingBindingTailoringManyEnablingRulePass (0.00s)
=== RUN   TestScanSettingBindingWatchesTailoredProfile
=== PAUSE TestScanSettingBindingWatchesTailoredProfile
=== RUN   TestManualRulesTailoredProfile
=== PAUSE TestManualRulesTailoredProfile
=== RUN   TestHideRule
    main_test.go:750: Skipping non-critical test
--- SKIP: TestHideRule (0.00s)
=== CONT  TestScanTailoredProfileIsDeprecated
=== CONT  TestScanSettingBindingWatchesTailoredProfile
=== CONT  TestManualRulesTailoredProfile
=== CONT  TestSingleTailoredScanSucceeds
2026/03/11 17:33:17 Creating ProfileBundle test-manual-rules-tailored-profile
2026/03/11 17:33:23 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:23 TailoredProfile ready (READY)
=== NAME  TestScanTailoredProfileIsDeprecated
    common.go:521: Found ComplianceScan deprecated profile event: TailoredProfile test-tailored-profile-is-deprecated is deprecated and should be avoided. Please consider using another profile
2026/03/11 17:33:28 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:28 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:28 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:33 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:33 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:33 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
--- PASS: TestScanSettingBindingWatchesTailoredProfile (16.30s)
2026/03/11 17:33:38 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:38 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:38 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:43 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:43 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:43 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:48 ProfileBundle ready (VALID)
2026/03/11 17:33:48 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:48 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:53 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:53 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:33:53 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:33:58 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:58 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:33:58 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:03 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:03 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:34:03 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:08 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:08 ComplianceScan ready (DONE)
2026/03/11 17:34:08 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
--- PASS: TestScanTailoredProfileIsDeprecated (51.17s)
2026/03/11 17:34:13 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:13 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:18 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:18 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:23 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:23 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:28 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:28 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:33 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:33 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: AGGREGATING
2026/03/11 17:34:38 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:38 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: AGGREGATING
2026/03/11 17:34:43 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:48 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:49 ComplianceScan ready (DONE)
2026/03/11 17:34:53 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:54 ComplianceScan ready (DONE)
2026/03/11 17:34:54 All scans in ComplianceSuite have finished (manual-rules-test-node)
--- PASS: TestManualRulesTailoredProfile (97.12s)
2026/03/11 17:34:58 Waiting for run of test-tailoredprofile-master compliancescan (AGGREGATING)
2026/03/11 17:35:03 ComplianceScan ready (DONE)
2026/03/11 17:35:08 ComplianceScan ready (DONE)
--- PASS: TestSingleTailoredScanSucceeds (111.44s)
PASS

Assisted-by: Cursor

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Mar 11, 2026

@Anna-Koudelkova: This pull request references CMP-3879 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

First attempt to start with the refactor based on feature groups. Not complete as we have not finished porting all the test cases upstream and also I need some guidance on importance of the test cases.

This is heavily coming from the refactor doc, trying to cover the tailoring tests.

  1. Test case execution order
    One thing I have found out is that golang would go through all the test cases and when it is parallel, it puts it on hold and run the serial test cases one by one first and the parallel test cases gets to run last.

  2. Flags
    For the test purposes I have labeled 4 test cases as Critical based on Cursor suggestion - it does not reflect the real importance!
    I was unable to use the Critical flag simply as part of the name of the test case (to run all the critical test cases only), because some of the CRDs like ScanSettings etc. gets named after the name of the tests and it cannot handle the underscores ( I was using TestName_Critical at first) and fails.
    That is why I resulted into filtering the test cases using flag.Bool() which seems to be working now:

make e2e-tailoring-critical   E2E_CONTENT_IMAGE_PATH="quay.io/rh-ee-akoudelk/content_repo:testcontent"  E2E_BROKEN_CONTENT_IMAGE_PATH="ghcr.io/complianceascode/test-broken-content-ocp"  E2E_GO_TEST_FLAGS="-v -timeout 60m"
..
=== RUN   TestScanTailoredProfileIsDeprecated
=== PAUSE TestScanTailoredProfileIsDeprecated
=== RUN   TestScanTailoredProfileHasDuplicateVariables
   main_test.go:128: Skipping non-critical test
--- SKIP: TestScanTailoredProfileHasDuplicateVariables (0.00s)
=== RUN   TestSingleTailoredScanSucceeds
=== PAUSE TestSingleTailoredScanSucceeds
=== RUN   TestScanSettingBindingTailoringManyEnablingRulePass
   main_test.go:293: Skipping non-critical test
--- SKIP: TestScanSettingBindingTailoringManyEnablingRulePass (0.00s)
=== RUN   TestScanSettingBindingWatchesTailoredProfile
=== PAUSE TestScanSettingBindingWatchesTailoredProfile
=== RUN   TestManualRulesTailoredProfile
=== PAUSE TestManualRulesTailoredProfile
=== RUN   TestHideRule
   main_test.go:750: Skipping non-critical test
--- SKIP: TestHideRule (0.00s)
=== CONT  TestScanTailoredProfileIsDeprecated
=== CONT  TestScanSettingBindingWatchesTailoredProfile
=== CONT  TestManualRulesTailoredProfile
=== CONT  TestSingleTailoredScanSucceeds
2026/03/11 17:33:17 Creating ProfileBundle test-manual-rules-tailored-profile
2026/03/11 17:33:23 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:23 TailoredProfile ready (READY)
=== NAME  TestScanTailoredProfileIsDeprecated
   common.go:521: Found ComplianceScan deprecated profile event: TailoredProfile test-tailored-profile-is-deprecated is deprecated and should be avoided. Please consider using another profile
2026/03/11 17:33:28 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:28 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:28 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:33 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:33 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:33 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
--- PASS: TestScanSettingBindingWatchesTailoredProfile (16.30s)
2026/03/11 17:33:38 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:38 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:38 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:43 waiting ProfileBundle test-manual-rules-tailored-profile to become VALID (PENDING)
2026/03/11 17:33:43 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:43 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:48 ProfileBundle ready (VALID)
2026/03/11 17:33:48 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:48 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/03/11 17:33:53 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:53 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:33:53 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:33:58 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:33:58 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:33:58 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:03 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:03 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/03/11 17:34:03 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:08 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:08 ComplianceScan ready (DONE)
2026/03/11 17:34:08 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
--- PASS: TestScanTailoredProfileIsDeprecated (51.17s)
2026/03/11 17:34:13 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:13 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:18 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:18 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:23 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:23 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:28 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:28 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: RUNNING
2026/03/11 17:34:33 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:33 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: AGGREGATING
2026/03/11 17:34:38 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:38 waiting until suite manual-rules-test-node reaches target status 'DONE'. Current status: AGGREGATING
2026/03/11 17:34:43 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:48 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:49 ComplianceScan ready (DONE)
2026/03/11 17:34:53 Waiting for run of test-tailoredprofile-master compliancescan (RUNNING)
2026/03/11 17:34:54 ComplianceScan ready (DONE)
2026/03/11 17:34:54 All scans in ComplianceSuite have finished (manual-rules-test-node)
--- PASS: TestManualRulesTailoredProfile (97.12s)
2026/03/11 17:34:58 Waiting for run of test-tailoredprofile-master compliancescan (AGGREGATING)
2026/03/11 17:35:03 ComplianceScan ready (DONE)
2026/03/11 17:35:08 ComplianceScan ready (DONE)
--- PASS: TestSingleTailoredScanSucceeds (111.44s)
PASS

Assisted-by: Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Mar 11, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Mar 11, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Anna-Koudelkova

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 11, 2026

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1105-90e2feae96931509d81cd43dbe64c9d41f798b59

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

approved do-not-merge/work-in-progress jira/valid-reference

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Anna-Koudelkova @openshift-ci-robot