Proof of concept: plugin architecture based on event bus + plugin listeners, using go-plugin

[metacosm]

What is the purpose of this change? What does it change?

This proof of concept takes the work accomplished in #1173 and changes the underlying mechanism to use hashicorp/go-plugin instead, using plain RPC.

Was the change discussed in an issue?

No.

How to test changes?

• Go to a directory outside of your $GOPATH (unless you know how to deal with go modules installed in $GOPATH)
• git clone https://github.com/metacosm/odo-tracer-listener.git
• cd odo-tracer-listener
• git checkout go-plugin
• go build -o tracer.listener.odo listener.go
• mkdir -p ~/.odo/plugins/ (if you don't already have that dir)
• cp tracer.listener.odo ~/.odo/plugins/
• Build this PR and issue an odo command that uses the new oc command pattern (any service, catalog or link/unlink commands, only these commands are automatically wired to dispatch events)
• You should see some plugin-related debugging messages
• Check ~/.odo/plugins/tracer.log to see the logging performed by the tracer plugin.

Note: It might happen that the command you issued failed with an error and you might need to check that the plugin process is not still running: ps aux | grep tracer and then killall tracer.listener.odo. Basically, plugins are "managed" and need to be cleaned-up before the application exits. This is done by a deferred call in the main odo function. However, calls to os.Exit don't run deferred functions. I have added some code to perform plugins clean up before some calls to os.Exit but might have missed some, which might lead dangling plugin processes running (basically a server process). A proper solution to this issue will have to be devised.

[codeclimate]

Code Climate has analyzed commit c504336 and detected 15 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1
Bug Risk	1
Style	13

View more on Code Climate.

[AppVeyorBot]

✅ odo build 1184 completed and artifacts can be found here (for commit a36c516cd6 by @metacosm)

[codecov]

Codecov Report

Merging #1235 into master will decrease coverage by 0.16%.
The diff coverage is 32.55%.

@@            Coverage Diff             @@
##           master    #1235      +/-   ##
==========================================
- Coverage   42.53%   42.36%   -0.17%     
==========================================
  Files          40       41       +1     
  Lines        5050     5136      +86     
==========================================
+ Hits         2148     2176      +28     
- Misses       2687     2741      +54     
- Partials      215      219       +4

Impacted Files	Coverage Δ
pkg/odo/util/completion/completion.go	0% <ø> (ø)	⬆️
pkg/config/config.go	48.83% <0%> (-2.75%)	⬇️
pkg/odo/util/cmdutils.go	4.87% <0%> (-0.13%)	⬇️
pkg/odo/events/bus.go	40.57% <40.57%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update caea0b7...c504336. Read the comment docs.

[AppVeyorBot]

✅ odo build 1185 completed and artifacts can be found here (for commit 3cb62c9995 by @metacosm)

[AppVeyorBot]

✅ odo build 1186 completed and artifacts can be found here (for commit 6e26e38e21 by @metacosm)

[AppVeyorBot]

✅ odo build 1187 completed and artifacts can be found here (for commit f67647ff1a by @metacosm)

[kadel]

I have a lot of questions about plugin mechanism:

• Have you considered using the same plugin system as kubectl?
• How will be plugins distributed and installed?
• What is the use case for plugins?
• Do you have any particular plugin in mind while implementing this?

My other concern is that we have quite a lot of fundamental problems mainly with interacting with S2I.
We definitely want some kind of plugin mechanism for odo in the future. But we need to have a stable foundation that we can build on. Odo won't be useful without stable core functionality. If it is not useful, no-one is going to use odo or its plugins :-(

[metacosm]

I have a lot of questions about plugin mechanism:

* Have you considered using the same plugin system as kubectl?

There is no plugin system in kubectl, just conventionally called external binaries. So you really cannot call that a plugin system as there is no real information exchange between the host and the "plugin".

* How will be plugins distributed and installed?

Binaries located in .odo/plugins/ right now, though it could be changed.

* What is the use case for plugins?

See: #1173
#1236
#1238

General use case: be able to extend odo without us having to implement everything people might want to do with it… :)

* Do you have any particular plugin in mind while implementing this?

As has been discussed several times already, the initial use case was supporting component CRD work without impacting the odo core.

My other concern is that we have quite a lot of fundamental problems mainly with interacting with S2I.
We definitely want some kind of plugin mechanism for odo in the future. But we need to have a stable foundation that we can build on. Odo won't be useful without stable core functionality. If it is not useful, no-one is going to use odo or its plugins :-(

Understood, though none of these issues are being worked on at the moment, at least, that I can see… The biggest obstacle I see right now is not moving fast enough on small things that should be done in a couple of days but are blocked by a review process that takes way too long.

[kadel]

I have a lot of questions about plugin mechanism:

* Have you considered using the same plugin system as kubectl?

There is no plugin system in kubectl, just conventionally called external binaries. So you really cannot call that a plugin system as there is no real information exchange between the host and the "plugin".

I agreed that this is not an ideal approach but on the other hand, it is easy to do, enough for a lot of use cases, and more importantly, it could mean that kubectl plugins will be compatible with odo. Which would be really nice.

* How will be plugins distributed and installed?

Binaries located in .odo/plugins/ right now, though it could be changed.

I know, but how the user will install it? Are we going to guide users in the docs to copy binaries to this directory? Are we going to provide some easier mechanism like odo plugin install etc..

My other concern is that we have quite a lot of fundamental problems mainly with interacting with S2I.
We definitely want some kind of plugin mechanism for odo in the future. But we need to have a stable foundation that we can build on. Odo won't be useful without stable core functionality. If it is not useful, no-one is going to use odo or its plugins :-(

Understood, though none of these issues are being worked on at the moment, at least, that I can see…

You are very welcomed to help us solve those issues.

The biggest obstacle I see right now is not moving fast enough on small things that should be done in a couple of days but are blocked by a review process that takes way too long.

Agreed, but I'm afraid that adding big changes that are not properly defined beforehand are going to make things even worse.
This is one of the reasons review process takes way too long.

[metacosm]

There is no plugin system in kubectl, just conventionally called external binaries. So you really cannot call that a plugin system as there is no real information exchange between the host and the "plugin".

I agreed that this is not an ideal approach but on the other hand, it is easy to do, enough for a lot of use cases, and more importantly, it could mean that kubectl plugins will be compatible with odo. Which would be really nice.

The only way kubectl plugins would work with odo was if odo had the same commands. As far as I understand it, at least: basically, if kubectl detects a properly named binary in the path, it will run that binary if its name matches the input command name. It would be easy to support this in odo as well, I think. That wouldn't help much, though…

Binaries located in .odo/plugins/ right now, though it could be changed.

I know, but how the user will install it? Are we going to guide users in the docs to copy binaries to this directory? Are we going to provide some easier mechanism like odo plugin install etc..

Could be an option, yes. You could imagine pulling a plugin from github and compiling it on the fly to install it.

My other concern is that we have quite a lot of fundamental problems mainly with interacting with S2I.
We definitely want some kind of plugin mechanism for odo in the future. But we need to have a stable foundation that we can build on. Odo won't be useful without stable core functionality. If it is not useful, no-one is going to use odo or its plugins :-(

Understood, though none of these issues are being worked on at the moment, at least, that I can see…

You are very welcomed to help us solve those issues.

That's not where my expertise lies, though, so I'm trying to help where I can help.

Agreed, but I'm afraid that adding big changes that are not properly defined beforehand are going to make things even worse.
This is one of the reasons review process takes way too long.

The point of this work (as is #1161) is to foster the discussion and see how things could work, because it's easier to reason on concrete examples than abstract ideas. Also, because restrictions might only become apparent after actually trying to implement a concept. However, if no one looks at it then it's just work for nothing.

[AppVeyorBot]

✅ odo build 1291 completed and artifacts can be found here (for commit 56ca095799 by @metacosm)

[cmoulliard]

supporting component CRD work without impacting the odo core.

another use cases could be to :

• audit, log what odo is doing
• scaffold a project (spring boot, vertx, tornthail, ....)
• export resources : helm chart, openshift template, ...

[AppVeyorBot]

✅ odo build 1350 completed and artifacts can be found here (for commit 792c7ac05f by @metacosm)

[AppVeyorBot]

✅ odo build 1391 completed and artifacts can be found here (for commit bf5448a365 by @metacosm)

[AppVeyorBot]

✅ odo build 1423 completed and artifacts can be found here (for commit e96682a99b by @metacosm)

[anmolbabu]

Extremely difficult to review the PR(very huge PR beyond the size I could think of for a PR ;) )...
Can we plan a session where you can walk the reviewers through the code?

[AppVeyorBot]

✅ odo build 1587 completed and artifacts can be found here (for commit 3f1deece37 by @metacosm)

[openshift-ci-robot]

@metacosm: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@metacosm: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-core	c504336	link	/test e2e-core
ci/prow/e2e-scenarios	c504336	link	/test e2e-scenarios

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[girishramnani]

This PR as of now needs alot of work due to the major changes in odo. It would be better if a new PR using the current master is created at a later point when plugins become priority.

Hence closing the PR