Add odo pipelines comands which generate GitOps pipelines resources #3387

wtam2018 · 2020-06-19T01:32:43Z

What type of PR is this?
/kind feature

Why

Summary

As a developer, after I'm happy trying out my code in inner-loop, I would want to manage deployments across multiple stages/environments from Git.

To do so, I would need tooling assitance to setup my environment and directories for Git-driven workflows in a standardized way.

As a developer ..

I want to be able to develop my application so that it can be CI/CD'ed by pipelines using the build/deploy guidance in devfile (devfile-awareness is not yet implemented).
I want to be able to review all Kubernetes resources with my team or organization before changes are applied to the cluster.
When no pipelines infrastructure currently exists, I want to be able to "bootstrap" my pipelines by generating basic CI/CD environment resources and my application deployment in that pipelines.
I do not need admin privileges to run pipelines commands. Administrators install all prerequisites on my cluster.

Prerequisites

Install the following as an admin :

Sealed Secrets

Please follow guidance in https://github.com/rhd-gitops-example/docs/tree/master/journey/day1#prerequisites for installing these on an OpenShift cluster.

In the near future, we are considering having a single operator to configure your cluster with these.

How

This enhancements provides the commands which generate Tekton, Kubernetes and Argo resources ("yamls") locally on the filesystem.
Users can then review the generated resources and push them to Git repository.
Once pushed to Git, the triggered PR webhook validates changes using a Tekton Pipeline.
Upon push to master, Argo takes care of deploying the relevant resources.

odo pipelines includes following sub-commands:

odo pipelines init

This command sets up "Gitops'd" CI/CD environments for you. Note, it does not create your first application and other environments, unlike the boostrap command. It outputs resources yaml files, kustomization files, and Manifest to filesystem.

odo pipelines bootstrap

The Bootstrap command creates default environments for your initial application.
It outputs resources yaml files, kustomization files, and pipelines.yaml manifest file, without making any changes to the cluster.

The following resources are written to the user's workspace in the local filesystem.

CI/CD environment with pipelines and resources.
ArgoCD environment manifests.
Dev environment manifests with an application/service.
Stage environment manifests.

No changes are made to the cluster.

odo pipelines environment

This command creates a new environment in an existing GitOps setup.
It outputs resources yaml files, kustomization files, and updated Manifest to filesystem, without making any changes to the cluster.

odo pipelines service

This command adds a new service to an existing environment. Services are logically grouped by applications. This command will create an application for the new service if the target application does not exist. It outputs resources yaml files, kustomization files, and updates the Gitops manifest locally on the filesystem.

No changes are made to the cluster.

odo pipelines webhook

Webhook commands create/delete/list webhooks for the Gitops manage source code repository as well as the service source code repository.

More documentation can be found here.
https://github.com/rhd-gitops-example/docs/tree/master/commands#odo-pipelines-commands
The odo pipelines command is currently hidden under experimental mode.

Fixes #3383

Testing this end-to-end

Note, the above commands generate manifests but do not apply anything on the cluster. To setup the webhook on your cluster so that you Git webhooks can talk to the cluster, you could need to do a

kubectl apply -k environments/*

In addition to Sealed Secrets, the following pre-reqs are required to set up for end-to-end testing:

Please follow guidance in https://github.com/rhd-gitops-example/docs/tree/master/journey/day1#prerequisites for installing these on an OpenShift cluster.

…edhat.com>

Co-authored-by: Ishita Sequelira <[email protected]>

- Added functions and unit tests to generate eventlistener - Added functions and unit tests to generate route - Appended the generated eventlistener and route to outputs slice.

- Imported v1alpha1 using glide get --strip-vendor

- Used fmt.Sprintf() to replace the repo name in filter

Delete accidentally added binary

- Added functions and test cases to create 1) deploy-from-source-task 2) deploy-using-kubectl-task 3) github-status-task Co-Authored-By: Gagan Hegde <[email protected]>

Co-authored-by: gaganeggday <[email protected]>

Co-authored-by: Gagan Hegde <[email protected]>

- Removed stuttering function names in tasks,routes and eventlisteners package. - Objects in task_tests are replaced by functions to improve readability.

* added two flags github-token and quay-io-auth-json * made changes in the flag description quayIOAuthFileName * added logic to get the github token filename and auth.json file name * made the requested changes mentioned in the PR * made the requested changes mentioned in the PR and removed the unwanted comments * created 4 required-flags quay-username, github-token, dockerconfigjson,base-repository * Update test cases for creating opaque secret * made the required changes mentioned in the PR * added two flags github-token and quay-io-auth-json * Add functions to generate Tekton task objects - Added functions and test cases to create 1) deploy-from-source-task 2) deploy-using-kubectl-task 3) github-status-task Co-Authored-By: Gagan Hegde <[email protected]> * made changes in the flag description quayIOAuthFileName * added logic to get the github token filename and auth.json file name made the requested changes mentioned in the PR made the requested changes mentioned in the PR and removed the unwanted comments created 4 required-flags quay-username, github-token, dockerconfigjson,base-repository Update test cases for creating opaque secret made the required changes mentioned in the PR * Replace objects in task_tests with suitable assertive functions - Removed stuttering function names in tasks,routes and eventlisteners package. - Objects in task_tests are replaced by functions to improve readability. * Add secret name as parameter to Generate function * pass ServiceAccount as object to createRoleBinding * pass ServiceAccount as object to createRoleBinding * Rework the new code around the base branch. Explicitly marshal the yaml to os.Stdout for now. Co-Authored-By: ishitasequeira <[email protected]> * made the required changes mentioned in the PR * removed the functions which were not required from bootstrap.go * removed the unwanted test case from bootstrap.go * Added the test file < odo/pkg/pipelines/bootstrap_test.go> * Test case corrected and moved to < /pkg/odo/cli/pipelines/bootstrap_test.go> * made all the changes asked in PR * pass a signel struct to Bootstrap() Co-authored-by: Ishita Sequeira <[email protected]> Co-authored-by: Chetan Banavikalmutt <[email protected]> Co-authored-by: gaganeggday <[email protected]> Co-authored-by: Kevin McDermott <[email protected]>

This generates the names for the namespaces and provides them when generating the resources. Co-authored-by: William Tam <[email protected]>

* Add functions to generate buildah task - Added functions to generate buildah task object - Added test cases for the corresponding functions * Replaced steps object by steps assertive functions * Fix typo in tasks_tests file Co-authored-by: William Tam <[email protected]>

sbose78 · 2020-06-30T11:46:49Z

Why isn't it pushed to oc which is far more aligned with the persona for that kind of operations ?

Hi DV!

oc is more of a tool to do kubectl+ anything specific to OpenShift . We are planning to move developer abstractions outside of oc ( example, oc new-app-ish workflows make more sense in odo ... ).

In this case, odo pipelines .... is an abstraction to generate and manage multi-stage pipelines for apps & components.

kadel · 2020-06-30T11:47:23Z

@bigkevmcd I think that there is some misunderstanding on both sides.

What might help is if you record a short end-to-end demo showcasing and explaining how odo pipelines should be used by developers.

kadel · 2020-06-30T11:52:52Z

In this case, odo pipelines .... is an abstraction to generate and manage multi-stage pipelines for apps & components.

@sbose78 What you are saying about odo pipelines is right. Problem is that from my point of view the current implementation is not doing that.
It is setting up GitOps flow with pipelines, not just pipelines. This is where probably some of the contentions is coming from original idea for odo pipelines was probably something a little bit different -> #1238

sbose78 · 2020-06-30T12:13:11Z

It is setting up GitOps flow with pipelines, not just pipelines

Correct. This is not just associating a tekton pipeline with a component in the cluster, odo pipelines effectively lays things out everything ( tekton, pure Kube, Argo CRs..) locally for subsequent Git-driven activities.

This does make it different from #1238 .
We could still expose things like odo pipelines status though, but we are yet to get there.

vdemeester · 2020-06-30T13:11:13Z

It is setting up GitOps flow with pipelines, not just pipelines

Correct. This is not just associating a tekton pipeline with a component in the cluster, odo pipelines effectively lays things out everything ( tekton, pure Kube, Argo CRs..) locally for subsequent Git-driven activities.

This does make it different from #1238 .
We could still expose things like odo pipelines status though, but we are yet to get there.

Any reason for it being named odo pipelines then instead of odo gitops or odo pipelines gitops ? odo pipelines make sense for most of things but bootstraping the gitops environment should be "on its own" (odo bootstrap … maybe ? so you can think of bootstraping more type of environment).

chmouel · 2020-06-30T13:29:29Z

Agreed with vincent and others, as a developer if I think about what to expect with "odo pipeline" i think about some default tasks associated to my project type (ie: python project => python unit test, flask, pep8 etc...) that get bootstraped to a pipeline for us, not necessary having a GitOps via pipeline env bootstraped....

PS: Cleaning up this PR, rebasing commits, separating deps in one commit, fixes in other commit and implementations in others would go a long way for reviewers to help them figure out what's going on with this PR...

…rator

openshift-ci-robot · 2020-07-01T04:35:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign mik-dass
You can assign the PR to them by writing /assign @mik-dass in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbose78 · 2020-07-01T04:45:47Z

@wtam2018 , could we do the following to make the commits reviewable?

A clean commit with the dependencies bumped in ~~go.mod and go.sum~~ gilde.
A bump(*) commit with the vendor updates.
A commit with the new functionality

sbose78 · 2020-07-01T04:46:54Z

Any reason for it being named odo pipelines then instead of odo gitops or odo pipelines gitops ?

gitops is a buzz-word, hence avoided having that in the command signature :-)

…tor (#135) Update glide to reference Triggers 0.5.0.

* renamed flags app-repo-url to service-repo-url and app-webhook-secret to service-webhook-secret * renamed the app and service name of the bootstrapped application and service

vdemeester · 2020-07-01T08:56:30Z

gitops is a buzz-word, hence avoided having that in the command signature :-)

I don't really buy that 😝. GitOps has hype right now, but why would the "name of that thing" change over time ? (just like git or devops, … do not change over time, being hype or not). But my point wasn't really on using gitops per-se, but on using pipelines for an specific setup of pipeline (named gitops). We may want to have different option when we want to bootstrap the user project pipelines, and one of them could be gitops.

I don't know if odo has a similar concept of TEP (Tekton Enhancement Proposal) or enhancements (like OpenShift), but discussing odo pipelines and its relation to gitops on something like that (TEP, enhancements) would have been a better start to discuss the feature (we want to provide the user) and gather early feedback (aka before making a great amount of work). It would have save some time for all parties I feel 😉.

kadel · 2020-07-01T10:53:44Z

I don't know if odo has a similar concept of TEP (Tekton Enhancement Proposal) or enhancements (like OpenShift), but discussing odo pipelines and its relation to gitops on something like that (TEP, enhancements) would have been a better start to discuss the feature (we want to provide the user) and gather early feedback (aka before making a great amount of work). It would have save some time for all parties I feel 😉.

We actually have a similar process in place that we call Proposals.
They are placed in docs/proposals - https://github.com/openshift/odo/tree/master/docs/proposals directory.

This work started in isolation from odo repository as a PoC, and then it went a little bit too far without creating Proposal and the result is this big PR.

I definitely don't blame anyone here expect myself for not communicating this with gitops team sooner.

vdemeester · 2020-07-01T11:13:39Z

We actually have a similar process in place that we call Proposals.
They are placed in docs/proposals - https://github.com/openshift/odo/tree/master/docs/proposals directory.

Ah good to know 👍

I definitely don't blame anyone here expect myself for not communicating this with gitops team sooner.

Yeah, definitely not blaming anyone, just trying to see what we can do better in the future 😉 👼

sbose78 · 2020-07-01T15:52:43Z

@vdemeester Totally understand the concerns around naming ambiguity. We should address it !

@kadel I'm aware of the enhancement proposal process, we should absolutely put in one @bigkevmcd @wtam2018 @ishitasequeira @chetan-rns .

As a background, we didn't start out thinking that we are going to have it all in odo. Rather, we had a separate bootstrapping/init mechanism.

We've iterated multiple times to find out the right solution to have tooling & guidance:

w/without Tekton &
w/without Argo &
w/without the idea of devfiles for outer-loop information.

We eventually decided to have the outcome of the above experiments as an ( initially isolated ) command in odo because that's where everything comes together and we are gradually charting out an experience from inner-loop to outer-loop.

Are we on the right path, we don't know!
We'll need a lot of feedback from customers while we accomplish the larger vision of a smooth inner-loop to outer-loop transition thereby ensuring that this doesn't impact the odo story.

* Add new functionality to secrets to find the sealed-secrets controller. This allows configuration of the namespace for sealed-secrets to be provided on the command-line. This also simplifies the configuration of the options, and standardises the Init and Bootstrap via an embedded struct.

wtam2018 · 2020-07-02T04:23:10Z

@sbose78

@wtam2018 , could we do the following to make the commits reviewable?

* A clean commit with the dependencies bumped in ~go.mod and go.sum~ gilde.

* A bump(*) commit with the vendor updates.

* A commit with the new functionality

I'd love to organize commits into vendor/dep and new features changes but I am not sure it is possible with such a big rebase. However, the only real feature changes are added in two packages.

pkg/pipelines
pkg/odo/cli/pipelines

As far as vendor files updates, vendor-update/glide update has been run before commits so the glide.yaml should reflect the correct states/changes of vendor/dependencies.

sbose78 · 2020-07-02T11:39:47Z

@wtam2018 Would be nice if all feature changes are collapsed into one commit.

bigkevmcd · 2020-07-02T11:58:06Z

@wtam2018 Would be nice if all feature changes are collapsed into one commit.

@sbose78 We should only do this when they're ready to review it, doing that would break the current work-in-progress.

wtam2018 · 2020-07-02T12:52:46Z

@wtam2018 Would be nice if all feature changes are collapsed into one commit.

@sbose78 We should only do this when they're ready to review it, doing that would break the current work-in-progress.

As scary as it may look, changes for the new features are in two packages:

pkg/pipelines
pkg/odo/cli/pipelines

sbose78 · 2020-07-02T13:08:01Z

Can we collapse those changes into 1 commit ?
Also, could you update the PR description ASAP please :) https://gist.github.com/sbose78/560ed43ef9da5bd07219bf6996a72c3c

* provide default (kube-system) for sealed-secret-ns flag * removed unintended/duplicated MarkFlagRequired()

* made gitops-webhook-secret flag optional * made sealed-secrets-ns flag optional and default to "kube-system"

openshift-ci-robot · 2020-07-07T03:04:32Z

@wtam2018: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/unit	`b46b6b9`	link	`/test unit`
ci/prow/v4.4-integration-e2e	`b46b6b9`	link	`/test v4.4-integration-e2e`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

wtam2018 and others added 30 commits March 3, 2020 10:30

implement tekton checks Co-authorized-by Ishita Sequelira <isequeir@r…

82bfe72

…edhat.com>

incoporate reiview feedbacks

40147fa

pass array of crd name strings

3c123ea

Restructure code and add unit test

759f19b

Co-authored-by: Ishita Sequelira <[email protected]>

Kill commented import

14a2b55

Co-authored-by: Ishita Sequelira <[email protected]>

Add eventlistener and route

feb3139

- Added functions and unit tests to generate eventlistener - Added functions and unit tests to generate route - Appended the generated eventlistener and route to outputs slice.

Import dependency for pipeline v1alpha1

9a72adf

- Imported v1alpha1 using glide get --strip-vendor

Replace github repo name in filter string

2da2e8d

- Used fmt.Sprintf() to replace the repo name in filter

Rename baseRepo param to githubRepo param

99a8f53

fix typos, rename interface, use t.Efrror in table test

b2f67d3

add error wrapping to errors returned by apis

766c5e4

Delete pipelines

0337e2e

Delete accidentally added binary

Add functions to generate Tekton task objects

d3d767c

- Added functions and test cases to create 1) deploy-from-source-task 2) deploy-using-kubectl-task 3) github-status-task Co-Authored-By: Gagan Hegde <[email protected]>

initial commit for Gagan

93c5d14

Co-authored-by: gaganeggday <[email protected]>

initial sumibt oc_adm.go, clean up service bindings.go

bb0abcd

rename format parameter name to avoid name collision

4ef3949

polish up oc_adm.go and add unit test

b416a94

Add cluster role "edit" creation

ffdb0d7

Co-authored-by: Gagan Hegde <[email protected]>

Replace objects in task_tests with suitable assertive functions

8437966

- Removed stuttering function names in tasks,routes and eventlisteners package. - Objects in task_tests are replaced by functions to improve readability.

create functions and get rid of the block comments

42fe854

incorporate review feedbacks

bcc8369

rename formal arguments from xxxName to name

73b921a

Add secret name as parameter to Generate function

946c629

pass ServiceAccount as object to createRoleBinding

2eacb03

pass ServiceAccount as object to createRoleBinding

f31ae04

pass slice of key/value pairs instead of strings (#13)

8cb8480

minor flag rename, update description (#12)

695d048

Add initial namespace creation logic. (#8)

18c6675

This generates the names for the namespaces and provides them when generating the resources. Co-authored-by: William Tam <[email protected]>

* lock github.com/tektoncd/triggers to v0.5.0 to work with Tekton ope…

65009be

…rator

wtam2018 and others added 2 commits July 1, 2020 08:39

Lock Tekton Triggers version to v0.5.0 so that it can work with opera…

d163618

…tor (#135) Update glide to reference Triggers 0.5.0.

Gitops issue 201- Changes in bootstrap command (#133)

c47f33d

* renamed flags app-repo-url to service-repo-url and app-webhook-secret to service-webhook-secret * renamed the app and service name of the bootstrapped application and service

Merge branch 'pipelines_feature_dev' into pipelines_feature

3557660

bigkevmcd and others added 5 commits July 2, 2020 14:51

Revert to pre-1.14 compatibility.

f05a2c9

* fix unresolved merge conflicts that made into commit.

4c2409d

* provide default (kube-system) for sealed-secret-ns flag * removed unintended/duplicated MarkFlagRequired()

Add --sealed-secrets-ns flag to "odo pipelines service add"

74dc1b6

Fix init command:

a461b1a

* made gitops-webhook-secret flag optional * made sealed-secrets-ns flag optional and default to "kube-system"

Fix broken pipelines init cmd test

b46b6b9

wtam2018 closed this Sep 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add odo pipelines comands which generate GitOps pipelines resources #3387

Add odo pipelines comands which generate GitOps pipelines resources #3387

wtam2018 commented Jun 19, 2020 •

edited

Loading

sbose78 commented Jun 30, 2020

kadel commented Jun 30, 2020

kadel commented Jun 30, 2020 •

edited

Loading

sbose78 commented Jun 30, 2020

vdemeester commented Jun 30, 2020

chmouel commented Jun 30, 2020 •

edited

Loading

openshift-ci-robot commented Jul 1, 2020

sbose78 commented Jul 1, 2020 •

edited

Loading

sbose78 commented Jul 1, 2020

vdemeester commented Jul 1, 2020

kadel commented Jul 1, 2020

vdemeester commented Jul 1, 2020

sbose78 commented Jul 1, 2020

wtam2018 commented Jul 2, 2020 •

edited

Loading

sbose78 commented Jul 2, 2020

bigkevmcd commented Jul 2, 2020

wtam2018 commented Jul 2, 2020

sbose78 commented Jul 2, 2020

openshift-ci-robot commented Jul 7, 2020 •

edited

Loading

Add odo pipelines comands which generate GitOps pipelines resources #3387

Add odo pipelines comands which generate GitOps pipelines resources #3387

Conversation

wtam2018 commented Jun 19, 2020 • edited Loading

Why

Summary

As a developer ..

Prerequisites

How

odo pipelines init

odo pipelines bootstrap

odo pipelines environment

odo pipelines service

odo pipelines webhook

Testing this end-to-end

sbose78 commented Jun 30, 2020

kadel commented Jun 30, 2020

kadel commented Jun 30, 2020 • edited Loading

sbose78 commented Jun 30, 2020

vdemeester commented Jun 30, 2020

chmouel commented Jun 30, 2020 • edited Loading

openshift-ci-robot commented Jul 1, 2020

sbose78 commented Jul 1, 2020 • edited Loading

sbose78 commented Jul 1, 2020

vdemeester commented Jul 1, 2020

kadel commented Jul 1, 2020

vdemeester commented Jul 1, 2020

sbose78 commented Jul 1, 2020

wtam2018 commented Jul 2, 2020 • edited Loading

sbose78 commented Jul 2, 2020

bigkevmcd commented Jul 2, 2020

wtam2018 commented Jul 2, 2020

sbose78 commented Jul 2, 2020

openshift-ci-robot commented Jul 7, 2020 • edited Loading

wtam2018 commented Jun 19, 2020 •

edited

Loading

kadel commented Jun 30, 2020 •

edited

Loading

chmouel commented Jun 30, 2020 •

edited

Loading

sbose78 commented Jul 1, 2020 •

edited

Loading

wtam2018 commented Jul 2, 2020 •

edited

Loading

openshift-ci-robot commented Jul 7, 2020 •

edited

Loading