Feature: Ordered Install/Upgrade

[mattfarina]

There have been discussions about adding the ability for Helm to do an ordered installation where different parts of the order are applied to Kubernetes at different times.

Currently, Helm sends all the resources (except CRDs) at one time. When there are subcharts, which are embedded in the chart, those are pulled into a shared sandbox, rendered together, and sent at once. Helm does order the resources by type as we have experience that this has mattered in some cases. The reason Helm sends everything at once is that controllers should eventually reconcile and make everything consistent.

This does not always work for applications, in practice.

Configuration management tools can be used to install different parts of an application separately to deal with this problem. But, this does not work for general distribution of applications as charts where end users can helm install an application.

The proposal is to do this with multiple times Helm sends resource to Kubernetes.

There are currently two ideas being discussed:

1. The H4HIP: Helm Sequencing Proposal (H4HIP: Helm Sequencing Proposal #373) - where there are "layers" in a chart that are sent at different times. There are trade-offs with this effort. For example:
   • A single chart can have different parts that are sent at different times. This means you could have a database, caching server, and application server in a single chart (i.e. no subcharts) and they could be installed in an ordered manner.
   • What happens when subcharts introduce layers of their own and there is one large shared sandbox? How will the calculated layers work together and be easily debugged? There be dragons in this area.
   • Individual components cannot be upgraded separately. This is good if you want to disallow that but bad if you want to allow individual component upgrades.
2. Hypper was a project that built on Helm and had the concept of a shared dependency. sub-charts inserted this way were not bundled with the chart in the traditional way and each of these subcharts was installed installed before its parent in an ordered manner.
   • Each chart is self contained and can be reasoned about.
   • Applications that have an ordering of parts (rather than well formed individual components) need to be broken up into multiple charts in an un-natural manner
   • Individual subcharts can be upgraded individually (a trade off listed on the previous bullet set).
   • Parts can be easily reused (e.g., a database chart that's used by multiple applications). It's DRY in this sense.

There may be alternative solutions to this problem and we are happy to discuss them. Neither of these 2 proposals is a solution for all the things.

We are interested in feedback and questions on these two ideas.

[banjoh]

Applications that have an ordering of parts (rather than well formed individual components) need to be broken up into multiple charts in an un-natural manner

This statement sums the problem statement quite well

[banjoh]

Today, Application Distributors are more often than not composing applications from ready made charts from bitnami et. al or vendors using charts to distribute their applications. I do not have data to back this up, I'm mostly speaking from experience.

With that, we'll see Application Distributors need to package charts that require to be distributed as a single installable entities. Some of these applications need to be composed from external dependencies (databases, caches, object stores etc) that might be separate distinct charts or templates in a chart. Today, having these applications start up in a particular order isn't always a simple thing to achieve.

Here is a concrete, but simplified example. I have an application that will conditionally generate a self-singed certificate if one does not exist in a defined secret at start up. I would like to have cert-manager issue a cert before my application starts, so my application can use that instead. Once I distribute this application, I want to have my users run helm install foo-release foo-chart.

There are two strategies I can use to allow my application to wait for cert-manager to generate a certificate before starting

• Use init containers which at times need extra functionality present in external dependencies to support polling for readiness i.e I'd need to poll cert-manager custom resource status to ensure its completed generating my cert.
• Split their application into separate charts. Install them separately and --wait/--wait-for-jobs. At times, some scripting will be needed to really wait for all components to be ready. A DB migration for example needs some custom logic that's not exposed as a kubernetes status. In the cert-manager case, we'd need to check if my certificate has been issued. This approach also introduces a distribution challenge where the application distributor needs to send install instructions which they cannot ensure will be used correctly. Installations might end up being done in the wrong order, wrong versions etc. At this point we'd be venturing into the land of installer scripts.

With chart resource ordering, the problem I see isn't that getting resources ordered is an impossible task, its the difficulty of achieving it that is the issue. It becomes harder the further away an Application Distributor is to the applications being packaged, especially if they are composing different applications from different charts/projects.

[gjenkins8]

been thinking about this for the last few days (or weeks now).

First thought, is I think we should certainly support something with much better UX than using annotations as #373 proposes. For example, representing resource sets (or "layers") in separate directories (though I'm sure there is a better way out there than this example). Where the user can easily identify / support different resource sets, and their order without hunting through annotations in templates. I think #373 proposes annotations as that is perhap the only option for Charts V2 API as we have today.

Second thought, is that options 1. and two 2. ie. layers within a chart, or ordering sub-charts are not necessarily exclusive. But obviously they do occupy a large portion of the same solution space. The differences is perhaps whether Helm want to allow users to compose different charts into an "application". But that also might mean artificially breaking a chart into several pieces to support ordering (as describe above). There also might also be a middle group,

Third thought, extending on the above. Helm must consider upgrades as much as installs. Maintaining and upgrading software Day 2 is almost always harder than installing on Day 1. IMHO, a single package should always be subject to a single operation (install and upgrade) ie. all layers would be upgraded to a new chart version in a single Helm operation. Whereas, and IMHO especially if orchestrated by higher level software, a chart with ordered sub-chart dependencies could be updated progressively/independently e.g. db the blue/green the app layer.