Migrate from legacy agent-based GitLab's GitOps solution to Flux

[drustan]

Hello there,

Gitlab decided to switch to using Flux for GitOps stuff last year. I've put off making the change until now, but I'm struggling to get it to fit with my current setup.

My Gitlab project with the manifest files looks a lot like this : https://mygitlab.mycompany.net/iac/kubernetes

So far, this is what my manifests repository looks like :

.
├── .gitlab
│   └── agents
│       ├── agent01
│       │   └── config.yaml
│       └── agent01-dev
│           └── config.yaml
├── manifests
│   ├── development
│   │   ├── awx
│   │   │   ├── awx-operator.yaml
│   │   │   ├── deployment.yaml
│   │   │   ├── ns.yaml
│   │   │   ├── pvc.yaml
│   │   │   └── vault.yaml
│   │   ├── cert-manager
│   │   │   ├── cert-manager.yaml
│   │   │   ├── ns.yaml
│   │   │   ├── production-issuer.yaml
│   │   │   └── staging-issuer.yaml
│   │   ├── www
│   │   │   ├── cnp_egress.yaml
│   │   │   ├── cnp_ingress.yaml
│   │   │   ├── deployment.yaml
│   │   │   ├── ingress.yaml
│   │   │   ├── ns.yaml
│   │   │   ├── README.md
│   │   │   ├── sa.yaml
│   │   │   ├── svc.yaml
│   │   │   ├── vault.yaml
│   │   │   └── vhost-cm.yaml
│   │   ├── vault-secrets-operator
│   │   │   ├── crb.yaml
│   │   │   ├── helmchart.yaml
│   │   │   ├── ns.yaml
│   │   │   └── README.md
│   │   └── wp
│   │       ├── helmchart.yaml
│   │       ├── ns.yaml
│   │       └── vault.yaml
│   └── prod
│       ├── awx
│       │   ├── awx-operator.yaml
│       │   ├── deployment.yaml
│       │   ├── ns.yaml
│       │   ├── pvc.yaml
│       │   └── vault.yaml
│       ├── cert-manager
│       │   ├── cert-manager.yaml
│       │   ├── ns.yaml
│       │   ├── production-issuer.yaml
│       │   └── staging-issuer.yaml
│       ├── www
│       │   ├── cnp_egress.yaml
│       │   ├── cnp_ingress.yaml
│       │   ├── deployment.yaml
│       │   ├── ingress.yaml
│       │   ├── ns.yaml
│       │   ├── README.md
│       │   ├── sa.yaml
│       │   ├── svc.yaml
│       │   ├── vault.yaml
│       │   └── vhost-cm.yaml
│       ├── vault-secrets-operator
│       │   ├── crb.yaml
│       │   ├── helmchart.yaml
│       │   ├── ns.yaml
│       │   └── README.md
│       └── wp
│           ├── helmchart.yaml
│           ├── ns.yaml
│           └── vault.yaml
└── README.md

I followed the "migrate to flux" documentation : https://docs.gitlab.com/ee/user/clusters/agent/gitops/migrate_to_flux.html

I decided to start with my dev environment, so here's how I went about bootstrapping Flux:

flux bootstrap gitlab \
      --owner=iac \
      --hostname=mygitlab.mycompany.net \
      --repository=kubernetes \
      --branch=dev \
      --path=manifests/development/ \
      --deploy-token-auth

After running that, I noticed a new "flux-system" directory popped up in my "manifests/development" path:

.
├── manifests
│   ├── development
│   │   ├── […]
│   │   ├── flux-system
│   │   │   ├── gotk-components.yaml
│   │   │   ├── gotk-sync.yaml
│   │   │   └── kustomization.yaml
│   │   └── […]
│   └── prod
│       └── […]

Here's what's inside the "gotk-sync.yaml" file:

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: dev
  secretRef:
    name: flux-system
  url: https://mygitlab.mycompany.net/iac/kubernetes.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./manifests/development
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Here's what's inside the "kustomization.yaml" file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml

But, I'm stuck at this error when I check the logs:

$ kubectl get kustomizations.kustomize.toolkit.fluxcd.io -n flux-system
NAME AGE READY STATUS
flux-system 3d4h False kustomize build failed: accumulating resources: accumulation err='merging resources from './awx/ns.yaml': may not add resource with an already registered id: Namespace.v1.[noGrp]/awx.[noNs]': must build at directory: '/tmp/kustomization-1427210208/manifests/development/awx/ns.yaml': file is not directory

$ kubectl logs pod/kustomize-controller-7b7b47f459-27lp7 -n flux-system
[…]
{"level":"error","ts":"2024-03-11T15:17:09.258Z","msg":"Reconciliation failed after 283.638596ms, next try in 10m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"94475020-0abc-43cb-b4f5-769df4bc0ce1","revision":"dev@sha1:7fc99637be1b6cb229365827c4644461e29a9adf","error":"kustomize build failed: accumulating resources: accumulation err='merging resources from './awx/ns.yaml': may not add resource with an already registered id: Namespace.v1.[noGrp]/awx.[noNs]': must build at directory: '/tmp/kustomization-1198011735/manifests/development/awx/ns.yaml': file is not directory"}

What do you think I'm missing?

Thanks for your help