Skip to content
This repository was archived by the owner on Oct 10, 2023. It is now read-only.

[WIP]Implement packages installation APIs for cluster essential#4503

Open
Vandy-P wants to merge 1 commit into
mainfrom
cluster-essential
Open

[WIP]Implement packages installation APIs for cluster essential#4503
Vandy-P wants to merge 1 commit into
mainfrom
cluster-essential

Conversation

@Vandy-P
Copy link
Copy Markdown
Contributor

@Vandy-P Vandy-P commented Mar 19, 2023

What this PR does / why we need it

Implementing APIs for cluster essential installation

Which issue(s) this PR fixes

Fixes #

Describe testing done for PR

Release note


Additional information

Special notes for your reviewer

@Vandy-P Vandy-P requested a review from a team as a code owner March 19, 2023 16:33
Comment thread cluster-essentials/cluster_init.go Outdated
Comment thread cluster-essentials/cluster_init.go Outdated
Comment thread cluster-essentials/cluster_init.go Outdated
Comment thread cluster-essentials/clusterinit.go Outdated
Comment thread cluster-essentials/clusterinit.go Outdated
Comment thread cluster-essentials/clusterinit.go Outdated
Comment thread cluster-essentials/clusterinit.go Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 21, 2023

Codecov Report

Merging #4503 (40e6a97) into main (bb07e03) will decrease coverage by 0.92%.
The diff coverage is n/a.

❗ Current head 40e6a97 differs from pull request most recent head 734920f. Consider uploading reports for the commit 734920f to get more accurate results

@@            Coverage Diff             @@
##             main    #4503      +/-   ##
==========================================
- Coverage   49.77%   48.86%   -0.92%     
==========================================
  Files         453      483      +30     
  Lines       45424    47544    +2120     
==========================================
+ Hits        22612    23230     +618     
- Misses      20652    22099    +1447     
- Partials     2160     2215      +55     

see 35 files with indirect coverage changes

Copy link
Copy Markdown
Contributor

@codegold79 codegold79 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems pretty straight forward to me. I had a couple questions and minor, optional feedback. None block my approval, so I approve.

Comment thread cluster-essentials/clusterinit.go Outdated
Comment thread cluster-essentials/clusterinit.go Outdated
Comment thread util/carvelhelpers/kbld.go Outdated
Comment thread util/carvelhelpers/package.go Outdated
Comment thread util/carvelhelpers/package.go Outdated
Comment thread util/carvelhelpers/package.go Outdated
// Also creates missing directories if any
func SaveFile(filePath string, data []byte) error {
dirName := filepath.Dir(filePath)
if _, serr := os.Stat(dirName); serr != nil {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell, but looks like the serr information doesn't get passed along. Do you think that's all right? Maybe people don't need to know what serr is because you pass along merr later? Kind of strange, but I guess it can work.

@Vandy-P Vandy-P requested a review from a team March 23, 2023 06:06
@Vandy-P Vandy-P requested a review from prkalle as a code owner March 23, 2023 06:06
@Vandy-P Vandy-P requested a review from a team March 23, 2023 06:06
@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230323062110/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

Comment thread cluster-essentials/clusterinit.go Outdated
@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230323065946/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@Vandy-P Vandy-P force-pushed the cluster-essential branch from 7504a57 to 0d8bcf7 Compare March 23, 2023 11:36
@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230323114346/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@Vandy-P Vandy-P force-pushed the cluster-essential branch from 0d8bcf7 to 407b872 Compare March 23, 2023 16:22
@Vandy-P Vandy-P requested review from a team March 23, 2023 16:22
@Vandy-P Vandy-P force-pushed the cluster-essential branch from 407b872 to 25ad233 Compare March 23, 2023 16:48
@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230323170259/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230323190717/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230324063347/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

Comment thread cluster-essentials/test/package1/config/overlay.yaml Outdated
Comment thread cluster-essentials/clusterinit_test.go Outdated
@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230326174544/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230326181447/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@github-actions
Copy link
Copy Markdown

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4503/20230326183605/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

@danniel1205
Copy link
Copy Markdown
Contributor

Question: Why do we have go mod change in other modules ? (Those modules seem unrelated to your change)

@Vandy-P
Copy link
Copy Markdown
Contributor Author

Vandy-P commented Mar 27, 2023

@danniel1205 We created kube utility function [util/kube/kube_util.go ] which needs to use client-go v0.25.0 hence all the dependent packages which are using util package got updated with client-go version v0.25.0.

Comment thread pkg/client/cluster-essentials/clusterinit.go Outdated
Comment thread pkg/client/cluster-essentials/clusterinit.go Outdated
Comment thread pkg/client/cluster-essentials/clusterinit.go Outdated
Comment thread pkg/client/cluster-essentials/clusterinit.go Outdated
Comment thread pkg/client/cluster-essentials/clusterinit.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Copy link
Copy Markdown
Contributor

@avi-08 avi-08 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown

@rajathagasthya rajathagasthya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested some minor changes.

Comment thread cluster-essentials/pkg/client/fakes.go Outdated
Comment thread cluster-essentials/pkg/client/helpers.go Outdated
Comment thread cluster-essentials/pkg/client/helpers.go Outdated
Comment thread cluster-essentials/pkg/client/helpers.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
@sathyanarays
Copy link
Copy Markdown
Contributor

Should this be merged with main or feature/runtime-core?

Comment thread cluster-essentials/pkg/client/install_test.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/README.md Outdated
Comment thread cluster-essentials/pkg/client/install.go
}
var generatedManifestBytes []byte
for _, item := range packages {
packageDir := filepath.Join(bundleDir, item.Name())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm confused, you are saying this Install function, expects a imgpkg bundle path, but then here you say each package contains a .imgpkg directory, how can a bundle contain multiple .imgpkg directories? how does the structure of imgpkg bundle look like? may be tree structure in the documentation helps

Copy link
Copy Markdown
Contributor Author

@Vandy-P Vandy-P Apr 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

├── kapp-controller
│   ├── config
│   │   ├── overlays
│   │   │   ├── change-namespace.yaml
│   │   │   ├── update-clusterrole-with-psp.yaml
│   │   │   ├── update-configmap.yaml
│   │   │   ├── update-crds.yaml
│   │   │   ├── update-deployment.yaml
│   │   │   └── update-strategy-overlay.yaml
│   │   ├── upstream
│   │   │   └── kapp-controller.yaml
│   │   ├── values.star
│   │   └── values.yaml
│   └── .imgpkg
│   └── imgpkg.yml
└── secretgen-controller
├── config
│   └── secretgen-controller.yaml
└── .imgpkg
└── imgpkg.yml

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yharish991 each package is bundle itself

Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
@Vandy-P Vandy-P requested review from a team, rajathagasthya, sathyanarays and yharish991 May 1, 2023 20:36
Comment thread cluster-essentials/pkg/client/install.go Outdated
// This function attempts to install packages from the given imgpkg bundle path
// and waits until install is complete or the given timeout is reached, whichever
// occurs first.
func Install(ctx context.Context, config *rest.Config, clusterEssentialRepo, clusterEssentialVersion string, timeout time.Duration) error {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason why we want to call this clusterEssentialRepo, it is still a package bundle with some config in it, calling it package bundle makes more sense

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yharish991 , Its not exactly a package bundle but the repo where cluster essential package reside
Eg:

clusterEssentialRepo := "public.ecr.aws/abcd/cluster-essentials"
clusterEssentialVersion := "v0.0.1"

Comment thread cluster-essentials/pkg/client/helpers.go Outdated
if err != nil {
return err
}
err = validatePreInstall(ctx, clientSet, clusterEssentialVersion)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we first do pre-install validation and then download the bundle?

Comment thread cluster-essentials/pkg/client/helpers.go Outdated
Comment thread cluster-essentials/pkg/client/install.go Outdated
if !ok {
return fmt.Errorf("pre validation check failed for package %s, package is not a part of cluster essential", packageName)
}
// Get the existing cluster essential version
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you handle the non-cluster-essentials installed kapp/secretgen-controller?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for non-cluster-essential , prevalidation will fail at line 65
_, ok := deployment.Annotations[packageNameAnnotation]
// If annotation is not present then return error
if !ok {
return fmt.Errorf("pre validation check failed for package %s, package is not a part of cluster essential", packageName)
}

Comment thread cluster-essentials/pkg/client/install.go
@Vandy-P Vandy-P requested review from avi-08 and yharish991 May 4, 2023 10:50
@Vandy-P Vandy-P force-pushed the cluster-essential branch from 5c29595 to 82e5c83 Compare May 4, 2023 10:56
Signed-off-by: Vandana Pathak <pathakv@sjc-ubu-eng-127.vmware.com>
@Vandy-P Vandy-P force-pushed the cluster-essential branch from 82e5c83 to 734920f Compare May 4, 2023 11:02
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants