From 74c0f592b393b78167094107270216de462fb8e5 Mon Sep 17 00:00:00 2001 From: Shoubhik Bose Date: Wed, 3 Nov 2021 16:48:16 -0400 Subject: [PATCH 1/5] SHIP : Git Event-driven build executions --- ships/0025-event-driven-builds.md | 298 ++++++++++++++++++++++++++++++ 1 file changed, 298 insertions(+) create mode 100644 ships/0025-event-driven-builds.md diff --git a/ships/0025-event-driven-builds.md b/ships/0025-event-driven-builds.md new file mode 100644 index 0000000..3e9f37e --- /dev/null +++ b/ships/0025-event-driven-builds.md @@ -0,0 +1,298 @@ + + +--- +title: event-driven-builds +authors: + - "@sbose787" + +reviewers: + - TBD + +approvers: + - TBD + + +creation-date: yyyy-mm-dd +last-updated: yyyy-mm-dd +status: provisional|implementable|implemented|rejected|withdrawn|replaced +see-also: + - "/docs/proposals/this-other-neat-thing.md" +replaces: + - "/docs/proposals/that-less-than-great-idea.md" +superseded-by: + - "/docs/proposals/our-past-effort.md" +--- + +# Event-driven triggering of Shipwright Builds + + +## Release Signoff Checklist + +- [ ] Enhancement is `implementable` +- [ ] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] Graduation criteria for dev preview, tech preview, GA +- [ ] User-facing documentation is created in [docs](/docs/) + +## Open Questions [optional] + +1. Almost every user would need an exposed `Service`, how do we create a vendor-agnostic `Ingress` object ? + + +## Summary + +Trigger the execution of an image build based on a commit/push event from a relevant source code repository. + + + +## Motivation + +This enhancement proposal aims to API to enable users to express the intent to have their builds also triggered by events from a Git Repository. + +### Goals + +* Build a technology-agostic user experience for Git-triggered build executions. +* Add support for 'reacting' to events from Github and Gitlab repositories. + + + +### Non-Goals + +* Support events from generic git servers. The implementation should be extensible to support those in a non-breaking manner. +* Creation of the vendor-specific Ingress resources to expose the webhook URL. + + +## Proposal + + +### User Stories [optional] + +Detail the things that people will be able to do if this is implemented. Include as much detail as +possible so that people can understand the "how" of the system. The goal here is to make this feel +real for users without getting bogged down. + +#### Story 1 +As a user, I would like to define a `Build` and trigger the execution of the same upon pushes to my Git Repo + +#### Story 2 +As a user, I would like to define a `Build` and the execution of the same for pushes to any branch in that Git Repo. + +#### Story 3 +As a user, I would like to configure a secure webhook URL for triggering `Builds`. + + +### Implementation Notes + +Upon specification of the **new** API field `.spec.webhook`, the Shipwright Build Controller will do the needful to generate a webhook URL and +provide the information on the same in the `status` of the `Build` resource. + + +``` +spec: + ... + ... + webhook: + type: github + imageTagPolicy: short_sha # optional, allowed values: 'short_sha' , 'branch'. Defaults to 'branch'. + secretRef: # optional, will be genereated if not specified. + name: my-webhook-secret. +``` + +The `.status` sub-resource would contain the information + +``` +status: + webhook: + status: live + reason: "" # to be populated in case of error. + type: github + secretRef: # mandatory field, in-secure not an option. + name: _user_specified_or_generated + serviceRef: # kubernetes service which needs to be exposed. + name: _name_of_the_recieveing_webhook_traffic +``` + +Here's what a full Build resource would look like : +``` +kind: Build +metadata: + name: buildpack-nodejs-build +spec: + source: + url: https://github.com/shipwright-io/sample-nodejs + contextDir: source-build + strategy: + name: buildpacks-v3 + kind: ClusterBuildStrategy + output: + image: docker.io/${REGISTRY_ORG}/sample-nodejs:latest + credentials: + name: push-secret + webhook: + type: github + imageTagPolicy: short_sha # optional, allowed values: 'short_sha' , 'branch'. Defaults to 'branch'. + secretRef: # optional, will be genereated if not specified. + name: my-webhook-secret. + status: + ... + ... + webhook: + status: live + reason: "" # to be populated in case of error. + type: github + secretRef: # mandatory field, in-secure not an option. + name: _user_specified_or_generated + serviceRef: # kubernetes service which needs to be exposed. + name: _name_of_the_recieveing_webhook_traffic + ``` + + #### Pre-requisities + +The following items ( part of "Bill of materials" ) would need to be shipped with the Shipwright installation so that they could be consumed in the webhook-generation process: + +1. A `ClusterTriggerBinding` which exposes `$(body.head_commit.id)` and `$(body.ref)` from the webhook payload. + +``` +apiVersion: triggers.tekton.dev/v1alpha1 +kind: ClusterTriggerBinding +metadata: + name: github-shipwright-webhook +spec: + params: + - name: commit + value: $(body.head_commit.id) + - name: branch + value: $(body.ref) +``` + +2. A "Custom Tekton Task" controller with the following behaviour: + * _watches_ `Tekton` `Run` resources referencing Shipwright's `Build` resources. + * Expects the commit ID and branch name in the `params`. + * Based on the above information, the controller would generate the following: + * A `Build` with the revision and output image tag overwritten with the above information based on the optional `.spec.webhook.imageTagPolicy`. + * A `BuildRun` referencing the above `Build`. + + + +#### Webhook endpoint creation process + + +Upon creation of a `Build` resource by the user, the following would occur: + +1. The Shipwright Build Controller would create a `TriggerTemplate` that would map the information coming in from the event into the "parameters" + expected by the `Run` resource ie, the branch name & the commit ID. + + ``` + +### Generated by the `Build` reconciler. + +apiVersion: triggers.tekton.dev/v1alpha1 +kind: TriggerTemplate +metadata: + name: my-build-name +spec: + params: + - name: branch + - name: commit + resourceTemplates: + - apiVersion: tekton.dev/v1alpha1 + kind: Run + metadata: + generateName: build-execution- + spec: + ref: + apiVersion: shipwright.io/v1alpha1 + kind: Build + name: my-build-bame + timeout: 3000s + params: + - name: branch + value: $(tt.params.branch) + - name: commit + value: $(tt.params.commit) + ``` + +2. The Shipwright Build Controller would create a `EventListener` under-the-hood per build. + +``` +### Generated by the `Build` reconciler. + +apiVersion: triggers.tekton.dev/v1alpha1 +kind: EventListener +metadata: + name: name-of-the-build +spec: + serviceAccountName: pipeline + triggers: + - bindings: + - ref: github-shipwright-webhook + template: + name: name-of-the-build + +``` + + + +### Test Plan + +To be filled. + +### Release Criteria + +To be filled. + +#### Removing a deprecated feature [if necessary] + +N/A + +#### Upgrade Strategy [if necessary] + +N/A + +### Risks and Mitigations + +1. This feature opens up the possiblility of triggering Build executions for branches which weren't explicitly specified. + +This isn't a risk per se since repo committers would have the permissions needed to push a branch to the repo. Therefore, +as long as the image is tagged appropriately to indiciate that it is not built off 'main' ( or the branch specified iniially by the user in the `Build`), +this should not be a problem. + +While the enhancement could have been scoped to only deal with the branch/revision explicitly specified in the `Build` resource, +doing the same would have have excluded some key use cases where users push to a different branch before merging to 'main'. + +2. Exposing a webhook URL enables creation of pods ( ie, processes on the node ) by actors who may not necessarily have access to the cluster. + +This does open up an attack vector given the execution of the build is done using the configured service account. + +* Webhook-driven builds are being designed to be secure by default - the usage of a webhook secret is mandatory. +* The resulting `BuildRun` would be annotated with relevant metadata from the webhook event so that it's easy to trace the actor responsible for the trigger. + + +## Drawbacks + +1. Is this even a Shipwright concern or should this be the concern of a general-purpose CI ? +( to be filled ) + + + + +## Alternatives + +Similar to the `Drawbacks` section the `Alternatives` section is used to highlight and record other +possible approaches to delivering the value proposed by an enhancement. + +## Infrastructure Needed [optional] + +Use this section if you need things from the project. Examples include a new subproject, repos +requested, github details, and/or testing infrastructure. + +Listing these here allows the community to get the process for these resources started right away. + +## Implementation History + +Major milestones in the life cycle of a proposal should be tracked in `Implementation History`. + From 2cd63d1c2cb41296a18661052907246fa2bd378a Mon Sep 17 00:00:00 2001 From: Shoubhik Bose Date: Wed, 3 Nov 2021 16:54:25 -0400 Subject: [PATCH 2/5] Update 0025-event-driven-builds.md --- ships/0025-event-driven-builds.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/ships/0025-event-driven-builds.md b/ships/0025-event-driven-builds.md index 3e9f37e..3c26798 100644 --- a/ships/0025-event-driven-builds.md +++ b/ships/0025-event-driven-builds.md @@ -10,10 +10,24 @@ authors: - "@sbose787" reviewers: - - TBD + - "@gmontero" + - "@adamkaplan" + - "@ImJasonH" + - "@SaschaSchwarze0" + - "@HeavyWombat" approvers: - - TBD + - "@adamkaplan" + - "@SaschaSchwarze0" + + +title: webhook-validation authors: + +"@ImJasonH" reviewers: +"@gmontero" +"@zhangtbj" approvers: +"@qu1queee" +"@adamkaplan" creation-date: 2020-03-19 last-updated: 2020-03-19 status: provisional creation-date: yyyy-mm-dd @@ -27,12 +41,12 @@ superseded-by: - "/docs/proposals/our-past-effort.md" --- -# Event-driven triggering of Shipwright Builds +# Git Event-driven triggering of Shipwright Builds ## Release Signoff Checklist -- [ ] Enhancement is `implementable` +- [x] Enhancement is `implementable` - [ ] Design details are appropriately documented from clear requirements - [ ] Test plan is defined - [ ] Graduation criteria for dev preview, tech preview, GA From 6f7e6d6c4a2545aae6961142e56516256814d7b0 Mon Sep 17 00:00:00 2001 From: Shoubhik Bose Date: Wed, 3 Nov 2021 17:00:35 -0400 Subject: [PATCH 3/5] Update 0025-event-driven-builds.md --- ships/0025-event-driven-builds.md | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-) diff --git a/ships/0025-event-driven-builds.md b/ships/0025-event-driven-builds.md index 3c26798..0774c9e 100644 --- a/ships/0025-event-driven-builds.md +++ b/ships/0025-event-driven-builds.md @@ -21,24 +21,11 @@ approvers: - "@SaschaSchwarze0" -title: webhook-validation authors: - -"@ImJasonH" reviewers: -"@gmontero" -"@zhangtbj" approvers: -"@qu1queee" -"@adamkaplan" creation-date: 2020-03-19 last-updated: 2020-03-19 status: provisional -creation-date: yyyy-mm-dd -last-updated: yyyy-mm-dd -status: provisional|implementable|implemented|rejected|withdrawn|replaced -see-also: - - "/docs/proposals/this-other-neat-thing.md" -replaces: - - "/docs/proposals/that-less-than-great-idea.md" -superseded-by: - - "/docs/proposals/our-past-effort.md" +creation-date: 2021-11-03 +status: implementable + --- # Git Event-driven triggering of Shipwright Builds @@ -65,7 +52,8 @@ Trigger the execution of an image build based on a commit/push event from a rele ## Motivation -This enhancement proposal aims to API to enable users to express the intent to have their builds also triggered by events from a Git Repository. +This enhancement proposal aims to provide an API to enable users to express the intent of having their builds triggered by events from a Git Repository. + ### Goals @@ -123,7 +111,6 @@ status: webhook: status: live reason: "" # to be populated in case of error. - type: github secretRef: # mandatory field, in-secure not an option. name: _user_specified_or_generated serviceRef: # kubernetes service which needs to be exposed. @@ -157,7 +144,6 @@ spec: webhook: status: live reason: "" # to be populated in case of error. - type: github secretRef: # mandatory field, in-secure not an option. name: _user_specified_or_generated serviceRef: # kubernetes service which needs to be exposed. @@ -183,6 +169,8 @@ spec: value: $(body.ref) ``` +Similar `ClusterTriggerBinding`s need to be shipped for `Gitlab` and `BitBucket`. + 2. A "Custom Tekton Task" controller with the following behaviour: * _watches_ `Tekton` `Run` resources referencing Shipwright's `Build` resources. * Expects the commit ID and branch name in the `params`. @@ -249,7 +237,7 @@ spec: ``` - +And that's it, you may now go ahead and ### Test Plan From bb53631f156b417d2e2ed22c34b189a6737b4f4f Mon Sep 17 00:00:00 2001 From: Shoubhik Bose Date: Wed, 3 Nov 2021 17:02:43 -0400 Subject: [PATCH 4/5] typos --- ships/0025-event-driven-builds.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ships/0025-event-driven-builds.md b/ships/0025-event-driven-builds.md index 0774c9e..168356a 100644 --- a/ships/0025-event-driven-builds.md +++ b/ships/0025-event-driven-builds.md @@ -78,10 +78,10 @@ possible so that people can understand the "how" of the system. The goal here is real for users without getting bogged down. #### Story 1 -As a user, I would like to define a `Build` and trigger the execution of the same upon pushes to my Git Repo +As a user, I would like to define a `Build` and trigger the execution of the same upon pushes to my Git repository. #### Story 2 -As a user, I would like to define a `Build` and the execution of the same for pushes to any branch in that Git Repo. +As a user, I would like to define a `Build` and the execution of the same for pushes to any branch in that Git repository. #### Story 3 As a user, I would like to configure a secure webhook URL for triggering `Builds`. @@ -259,7 +259,7 @@ N/A 1. This feature opens up the possiblility of triggering Build executions for branches which weren't explicitly specified. -This isn't a risk per se since repo committers would have the permissions needed to push a branch to the repo. Therefore, +This isn't a risk per se since only repository committers would have the permissions needed to push branches. Therefore, as long as the image is tagged appropriately to indiciate that it is not built off 'main' ( or the branch specified iniially by the user in the `Build`), this should not be a problem. From c2c7404272269d48c89772cbfbeae31309167277 Mon Sep 17 00:00:00 2001 From: Shoubhik Bose Date: Sun, 28 Nov 2021 21:40:38 -0500 Subject: [PATCH 5/5] Do not support arbitrary brances --- ships/0025-event-driven-builds.md | 85 +++++++++++++++++++------------ 1 file changed, 53 insertions(+), 32 deletions(-) diff --git a/ships/0025-event-driven-builds.md b/ships/0025-event-driven-builds.md index 168356a..5e98ff9 100644 --- a/ships/0025-event-driven-builds.md +++ b/ships/0025-event-driven-builds.md @@ -58,7 +58,7 @@ This enhancement proposal aims to provide an API to enable users to express the ### Goals * Build a technology-agostic user experience for Git-triggered build executions. -* Add support for 'reacting' to events from Github and Gitlab repositories. +* Add support for 'reacting' to events from Git repositories. @@ -66,6 +66,7 @@ This enhancement proposal aims to provide an API to enable users to express the * Support events from generic git servers. The implementation should be extensible to support those in a non-breaking manner. * Creation of the vendor-specific Ingress resources to expose the webhook URL. +* Definition of what other forms of triggers may look like. ## Proposal @@ -73,17 +74,11 @@ This enhancement proposal aims to provide an API to enable users to express the ### User Stories [optional] -Detail the things that people will be able to do if this is implemented. Include as much detail as -possible so that people can understand the "how" of the system. The goal here is to make this feel -real for users without getting bogged down. #### Story 1 As a user, I would like to define a `Build` and trigger the execution of the same upon pushes to my Git repository. #### Story 2 -As a user, I would like to define a `Build` and the execution of the same for pushes to any branch in that Git repository. - -#### Story 3 As a user, I would like to configure a secure webhook URL for triggering `Builds`. @@ -97,9 +92,8 @@ provide the information on the same in the `status` of the `Build` resource. spec: ... ... - webhook: + trigger: type: github - imageTagPolicy: short_sha # optional, allowed values: 'short_sha' , 'branch'. Defaults to 'branch'. secretRef: # optional, will be genereated if not specified. name: my-webhook-secret. ``` @@ -108,13 +102,14 @@ The `.status` sub-resource would contain the information ``` status: - webhook: + trigger: status: live + type: github reason: "" # to be populated in case of error. secretRef: # mandatory field, in-secure not an option. name: _user_specified_or_generated serviceRef: # kubernetes service which needs to be exposed. - name: _name_of_the_recieveing_webhook_traffic + name: _name_of_the_recieving_webhook_traffic ``` Here's what a full Build resource would look like : @@ -133,21 +128,21 @@ spec: image: docker.io/${REGISTRY_ORG}/sample-nodejs:latest credentials: name: push-secret - webhook: + trigger: type: github - imageTagPolicy: short_sha # optional, allowed values: 'short_sha' , 'branch'. Defaults to 'branch'. secretRef: # optional, will be genereated if not specified. name: my-webhook-secret. status: ... ... - webhook: + trigger: + type: github status: live reason: "" # to be populated in case of error. secretRef: # mandatory field, in-secure not an option. - name: _user_specified_or_generated + : _user_specified_or_generated serviceRef: # kubernetes service which needs to be exposed. - name: _name_of_the_recieveing_webhook_traffic + name: _name_of_the_recieving_webhook_traffic ``` #### Pre-requisities @@ -160,13 +155,15 @@ The following items ( part of "Bill of materials" ) would need to be shipped wit apiVersion: triggers.tekton.dev/v1alpha1 kind: ClusterTriggerBinding metadata: - name: github-shipwright-webhook + name: github-shipwright-trigger spec: params: - name: commit value: $(body.head_commit.id) - name: branch value: $(body.ref) + - name: url + value: $(body.url) ``` Similar `ClusterTriggerBinding`s need to be shipped for `Gitlab` and `BitBucket`. @@ -175,9 +172,22 @@ Similar `ClusterTriggerBinding`s need to be shipped for `Gitlab` and `BitBucket` * _watches_ `Tekton` `Run` resources referencing Shipwright's `Build` resources. * Expects the commit ID and branch name in the `params`. * Based on the above information, the controller would generate the following: - * A `Build` with the revision and output image tag overwritten with the above information based on the optional `.spec.webhook.imageTagPolicy`. + * Read the `Build` in the namespace matching the `Run` CR's `.spec.ref` * A `BuildRun` referencing the above `Build`. - + + Sample `Run` CR the controller would be reacting to: + ``` + + - apiVersion: tekton.dev/v1alpha1 + kind: Run + metadata: + generateName: build-execution- + spec: + ref: + apiVersion: shipwright.io/v1alpha1 + kind: Build + name: build-cr-name + ``` #### Webhook endpoint creation process @@ -195,11 +205,12 @@ Upon creation of a `Build` resource by the user, the following would occur: apiVersion: triggers.tekton.dev/v1alpha1 kind: TriggerTemplate metadata: - name: my-build-name + name: build-cr-name spec: params: - name: branch - name: commit + - name: url resourceTemplates: - apiVersion: tekton.dev/v1alpha1 kind: Run @@ -208,14 +219,16 @@ spec: spec: ref: apiVersion: shipwright.io/v1alpha1 - kind: Build - name: my-build-bame + kind: Build + name: build-cr-name timeout: 3000s params: - - name: branch + - name: revision value: $(tt.params.branch) - name: commit value: $(tt.params.commit) + - name: url + value: $(tt.params.url) ``` 2. The Shipwright Build Controller would create a `EventListener` under-the-hood per build. @@ -226,14 +239,14 @@ spec: apiVersion: triggers.tekton.dev/v1alpha1 kind: EventListener metadata: - name: name-of-the-build + name: build-cr-name spec: serviceAccountName: pipeline triggers: - bindings: - ref: github-shipwright-webhook template: - name: name-of-the-build + name: build-cr-name ``` @@ -259,12 +272,9 @@ N/A 1. This feature opens up the possiblility of triggering Build executions for branches which weren't explicitly specified. -This isn't a risk per se since only repository committers would have the permissions needed to push branches. Therefore, -as long as the image is tagged appropriately to indiciate that it is not built off 'main' ( or the branch specified iniially by the user in the `Build`), -this should not be a problem. +The custom Tekton controller is responsible for ignoring any requests which would have originated from branches not explicitly +specified in the `Build` CR. -While the enhancement could have been scoped to only deal with the branch/revision explicitly specified in the `Build` resource, -doing the same would have have excluded some key use cases where users push to a different branch before merging to 'main'. 2. Exposing a webhook URL enables creation of pods ( ie, processes on the node ) by actors who may not necessarily have access to the cluster. @@ -276,10 +286,21 @@ This does open up an attack vector given the execution of the build is done usin ## Drawbacks -1. Is this even a Shipwright concern or should this be the concern of a general-purpose CI ? -( to be filled ) +1. This design wouldn't support changes to repository branches other than the one defined in the `Build` CR. + +As per discussion with the community, supporting the same was considered to be outside the scope of the initial +design of this feature. +In a future enhancement, we may consider adding something along the lines of the following +to properly handle image tagging based on handling of webhook triggers from multiple branches. +``` + webhook: + type: github + imageTagPolicy: short_sha # optional, allowed values: 'short_sha' , 'branch'. Defaults to 'branch'. + secretRef: # optional, will be genereated if not specified. + name: my-webhook-secret. +``` ## Alternatives