Deploying HelmRelease with sourceRef of GitRepository (private) fails

[naegleria]

Hi,
I have a setup where each application has their own Git repo (app1, app2, etc.), their own custom Helm chart and their own HelmRelease plus any additional .yaml files (wrapped by a kustomization.yaml) for the app and infrastructure separately (similarly to https://github.com/fluxcd/flux2-kustomize-helm-example/tree/main)
There is a separate repository called fluxcd, where I have bootstrapped K8s clusters, for example test cluster is bootstrapped to clusters/test and prod cluster to clusters/production. This is the general structure of the fluxcd repo:

.
├── clusters
│   ├── test
│   │   └── app1
│   │       ├── apps.yaml
│   │       ├── gitrepository.yaml
│   │       └── infrastructure.yaml
│   │   └── app2
│   │       ├── apps.yaml
│   │       ├── gitrepository.yaml
│   │       └── infrastructure.yaml
│   │   └── flux-system
│   │       ├── <files created during flux-bootstrap>
│   │   └── helm-repos
│   │       ├── bitnami-repo.yaml
│   │       ├── <some other repos>
│   └── production
│   │   └── app1
│   │       ├── apps.yaml
│   │       ├── gitrepository.yaml
│   │       └── infrastructure.yaml
│   │   └── flux-system
│   │       ├── <files created during flux-bootstrap>
│   │   └── <same thing as with test>

apps.yaml and infrastructure.yaml files are Kustomization files that reference their respective kustomization.yaml files inside the app repos. The HelmRelease yamls for infrastructure use the helm repositories defined in fluxcd project and as they are public repos, the infrastructure HelmRelease deployment works without issues, and the helm charts + other yamls get deployed as they should.

The setup for app deployment is as follows:
In app1 repo I have .helm/app1 directory containing Chart.yaml, values.yaml and templates, aka a normal helm chart structure.
In app1 repo I also have .kubernetes/apps/app1 directory containing a helmrelease.yaml and kustomization.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: app1
  namespace: app1
spec:
  interval: 8h
  releaseName: app1
  chart:
    spec:
      chart: ./helm/app1
      version: 0.*.*
      interval: 12h
      sourceRef:
        kind: GitRepository
        name: app1-repo
        namespace: flux-system
      valuesFiles:
        - ./helm/app1/values.yaml
  install:
    crds: Create
  upgrade:
    crds: CreateReplace

(According to documentation, version: parameter is ignored for GitRepository sourceRef, but I added it anyway, hope that didn't cause the issue.)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helmrelease.yaml

The referenced git repo is the gitrepository.yaml file defined in fluxcd repo in .clusters/test/app1:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: app1-repo
  namespace: flux-system
spec:
  interval: 1m
  ref:
    branch: main
  url: https://gitlab.server.url/app1/app1
  secretRef:
    name: app1-repo-deploy-token
  ignore: |
    /*
    !/kubernetes/
    !/helm/

So the source-controller authenticates with the app1 repo using the deploy token with read repo and registry rights.
And finally the apps.yaml file in fluxcd repo that is supposed to actually deploy the HelmRelease to K8s:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: app1
  namespace: flux-system
spec:
  dependsOn:
    - name: app1-mysql
  interval: 10m
  targetNamespace: app1
  sourceRef:
    kind: GitRepository
    name: app1-repo
  path: "./kubernetes/apps/app1"
  prune: true
  patches:
    - patch: | ...//...

The error message I am getting:
2024-03-13T10:54:07.596Z error Kustomization/flux-system.flux-system - Reconciliation failed after 576.420153ms, next try in 10m0s HelmRelease/app1/app1 dry-run failed (Invalid): HelmRelease.helm.toolkit.fluxcd.io "app1" is invalid: [spec.chart: Required value, spec.interval: Required value]

Now my assumption was that the app1 HelmRelease should be able to fetch the chart from the Git repo, as the reconciliation is done inside flux-system namespace, which can read app1 repo via the deploy token secret. Or is it necessary to define another gitrepository.yaml towards the same app1 repo just to be used for helm chart fetching, but with different parameters? Or maybe the issue is something else entirely, it's hard to tell as the error says it requires spec.chart and spec.interval values, which are defined.

Any help would be appreciated.

[souleb]

your helmrelease definition is the problem. The error says what's missing spec.chart: Required value, spec.interval: Required value. Also the chart name that you passed above is not a valid helmchart name.

[naegleria]

But how is it the problem? spec.chart and spec.interval are both present.
By valid helmchart name, do you mean .chart.spec.chart: ./helm/app1? According to documentation it is valid:
The .chart.spec.chart can either contain:
The name of the chart as made available by the HelmRepository (without any aliases), for example: podinfo
The relative path the chart can be found at in the GitRepository or Bucket, for example: ./charts/podinfo

[souleb]

Oh yes my bad. I can help debug this if you ping me in cncf slack.

[naegleria]

So this looks to be a bug on FluxCD side and does not have anything to do with HelmRelease itself, faulty or not.
To my understanding, the faults in configuration are reported as follows:

• Syntax and values issues in app1 kustomization (apps.yaml) are reported by app1 kustomization resource in K8s cluster in flux-system namespace
• Syntax and values issues in app1 helmrelease yaml file are reported by app1 helmrelease resource in K8s cluster in app1 namespace

In my case, flux-system kustomization.yaml is unable to create the app1 kustomization resource in flux-system with an error message that seems to indicate an issue with the HelmRelease:

$ kubectl --kubeconfig -n flux-system get kustomizations
NAME                AGE    READY   STATUS
app1-mailhog       3d      True    Applied revision: main@sha1:97009609a4efe8c15da2300c4a57cdd175f4a2ba
app1-mysql         3d20h   True    Applied revision: main@sha1:97009609a4efe8c15da2300c4a57cdd175f4a2ba
flux-system        4d22h   False   HelmRelease/app1/app1 dry-run failed (Invalid): HelmRelease.helm.toolkit.fluxcd.io "app1" is invalid: [spec.chart: Required value, spec.interval: Required value]...

The way I got around this issue was to simply move the content of apps.yaml kustomization to infrastructure.yaml kustomization file, so now infrastructure.yaml contains the previously created app1-mailhog, app1-mysql kustomizations as well as app1 kustomization, which was now also successfully created:

$ kubectl --kubeconfig -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
NAME                AGE   READY   STATUS
app1                19h   False   kustomize build failed: add operation does not apply: doc is missing path: "/spec/values/useenv/env/1/value": missing value
app1-mailhog      3d19h   True    Applied revision: main@sha1:97009609a4efe8c15da2300c4a57cdd175f4a2ba
app1-mysql        4d15h   True    Applied revision: main@sha1:97009609a4efe8c15da2300c4a57cdd175f4a2ba
flux-system       5d17h   True    Applied revision: main@sha1:d51c904b7a8fd62a2025daf854ea4f765f6ad05d

While there is an issue with patches add operation in app1 kustomization, the kustomization itself was created in the cluster, as it should be. After fixing the issues regarding patches add paths, the app1 kustomization resource reported READY as True:

$ kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
NAME               AGE     READY   STATUS
app1               11h     True    Applied revision: main@sha1:b37d92f1ea60d869ad523f5d264c8c1059b6c3b5
app1-mailhog       7d15h   True    Applied revision: main@sha1:b37d92f1ea60d869ad523f5d264c8c1059b6c3b5
app1-mysql         8d      True    Applied revision: main@sha1:b37d92f1ea60d869ad523f5d264c8c1059b6c3b5
flux-system        9d      True    Applied revision: main@sha1:1ffce5ff6116940b81ee895bf678e1422415646d

However, the app1 helmrelease resource in app1 namespace is reporting an issue:

$ kubectl -n ee-redcap get helmreleases.helm.toolkit.fluxcd.io
NAME           AGE     READY   STATUS
app1           7h51m   False   Helm upgrade failed for release app1/app1 with chart [email protected]+1: failed to create resource: Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type string
app1-mailhog   7d11h   True    Helm upgrade succeeded for release app1/app1-mailhog.v4 with chart [email protected]
app1-mysql     8d      True    Helm install succeeded for release app1/app1-mysql.v1 with chart [email protected]

This was just to illustrate that issues with HelmRelease are reported by HelmRelease resource, and not by flux-system kustomization resource.

Now back to the original issue, after app1 kustomization was successfully created, I removed the app1 yaml from infrastrcuture.yaml, which made flux remove app1 kustomization from the cluster, as it should. Next I added apps.yaml back with the app1 kustomization yaml, meaning I restored the original configuration that caused the error, but this time app1 kustomization resource was successfully created. Meaning that I could not reproduce the issue, maybe because there's some cache still present somewhere in flux-system namespace, I don't really know. Perhaps I can reproduce it by also removing infrastructure.yaml, and adding them back in the same order - first infrastructure.yaml and then apps.yaml, or perhaps a completely removal of flux-system is needed.

There seems to be other issues when it comes to flux-system kustomization reading files from the specified directory, for example right now I'm using inline JSON6902 format in apps.yaml file to apply patches, but I originally wanted to use path, like in this example and I created 2 helmrelease yaml files to apply patches from: appconf.yaml and sealedsecrets.yaml, but flux-system kustomization failed and gave this error:
flux-system 8d False kustomize build failed: accumulating resources: accumulation err='merging resources from './app1/sealedsecrets.yaml': may not add resource with an already registered id: HelmRelease.v2beta2.helm.toolkit.fluxcd.io/app1.app1': must build at directory: '/tmp/kustomization-1109167131/clusters/test/app1/sealedsecrets.yaml': file is not directory

So either this is a bug, or undocumented restrictions when it comes to reading .yaml files from the bootstrapped path.