How to deal with common code in multiple HelmReleases?

[joelperez98]

Hello everyone,
I want to add new annotations and tolerations to all my HelmReleases, starting first with some apps, then with all apps at dev and then with all apps on prod.

So my manager has told me to just do something like this for every app:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
...
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
...
spec:
...
  chart:
    spec:
       ...
      sourceRef:
        kind: HelmRepository
  values:
    ...
    query:
      ...
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "role"
                operator: In
                values:
                - "infra"
      tolerations:
      - key: "infra"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
    queryFrontend:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "role"
                operator: In
                values:
                - "infra"
      tolerations:
      - key: "infra"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
    compactor:
     ...
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "role"
                operator: In
                values:
                - "infra"
      tolerations:
      - key: "infra"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

Which is obviously not a clever solution as I need to repeat the affinity and tolerations every time for every pod.

After reading #2919, I've managed to setup a configMapGenerator for my HelmRelease, something like this:

configMapGenerator:
  - name: grafana-loki-values
    files:
      - values.yaml

However, I'm hitting the same problem as I have to hardcode the annotations every time for each pod. What I'd like to is to just have one single common-values.yaml file, like this:

common:
  affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "role"
                  operator: In
                  values:
                    - "infra"
    tolerations:
      - key: "infra"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

And then just use it from the required values.yaml files doing something similar to

gateway:
 ...
  affinity: ${AFFINITY}
  tolerations: ${TOLERATIONS}

For each HelmRelease. And it would be also nice if then I could easily path those values for each cluster/stage.

However, I don't manage to merge the values.yaml file with my common-values.yaml file and create a ConfigMap from those 2 bases files. I'm pretty sure there must be a way to achieve this. I don't know if configMapgenerator is suitable for that, or I'd have to use patch or maybe I'm missing something.

I have tried the behaviour: merge for the configMapGenerator but it's failing:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helmrelease.yaml
configMapGenerator:
  - name: grafana-loki-values
    behavior: merge
    files:
      - values.yaml
      - common-values.yaml

Error message:

flux build kustomization grafana-loki --path apps/base/grafana-loki --kustomization-file /Users/U1063621/sanofi/oneai/oneai-core-fleet-infra/base/services/grafana-loki.yaml > output.yaml
✗ kustomize build failed: merging from generator &{0x140006ac4e0 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"grafana-loki-values", Namespace:""} does not exist; cannot merge or replace

I don't have more ideas about what could I try or read.

Thank you for your time

[matheuscscp]

Use kustomize to have a base for the HelmRelease values.

[joelperez98]

I have one HelmRelease for each app, so the object configuration may change, so think about the Grafana stack, calico, kubecost and so on... So use Kustomize and then patch every HelmRelease is almost the same than just hardcoding every HelmRelease. Yes, I'll have a bit more flexibility but I'm still repeating a lot of code.

I've managed to do something which is near to work, but still does not work.

First, I'm creating a ConfigMap like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: common-values
  namespace: flux-system
data:
  AFFINITY: |
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "role"
                operator: In
                values:
                  - "infra"

Then, I use a substituteFrom in my Grafana-loki Kustomization file:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: grafana-loki
  namespace: flux-system
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/base/grafana-loki
  prune: true
  wait: true
  timeout: 5m0s
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
      - kind: ConfigMap
        name: common-values
  dependsOn:
    - name: crossplane-provider-aws

And then I create the HelmReleases from a values.yaml file, like this:

fullnameOverride: loki
loki:
.....
  affinity: ${AFFINITY}

However, when doing the flux build is not respecting the new lines! But yeah, I'm near to it