Kustomization Appears Stuck

[texasbobs]

We have a cluster with about 20 kustomizations running. All of them except one sync up to the latest commit in the repo. The one with the issue just says

kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io   monarch-cache
NAME            AGE    READY     STATUS
monarch-cache   190d   Unknown   reconciliation in progress

All of the other ones look normal

kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
NAME                          AGE    READY     STATUS
bifrost-telephone             91d    True      Applied revision: main/f21a6188484d0343335b13ecfdc0ad286316e
cert-manager                  190d   True      Applied revision: main/f21a6188484d0343335b13ecfdc0ad286316e
cns-pcc-defender              57d    True      Applied revision: main/f21a6188484d0343335b13ecfdc0ad286316e

The kustomization in question contains

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  creationTimestamp: null
  name: monarch-cache
  namespace: flux-system
spec:
  interval: 1h0m0s
  path: monarch-cache
  timeout: 5m0s
  retryInterval: 1m0s  
  prune: true
  force: true
  sourceRef:
    kind: GitRepository
    name: edge

The logs for the kustomization controller have no mention of trying to process this one. Any ideas on where to look for more info here?

The complete kustomization as it is in the cluster currently.

kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io   monarch-cache -o yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  creationTimestamp: "2022-03-03T06:00:05Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 3
  labels:
    kustomize.toolkit.fluxcd.io/name: kustomizations
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: monarch-cache
  namespace: flux-system
  resourceVersion: "178703422"
  uid: afbd901f-b335-485e-8dec-72c40f25bd8f
spec:
  force: true
  interval: 1h0m0s
  path: monarch-cache
  prune: true
  retryInterval: 1m0s
  sourceRef:
    kind: GitRepository
    name: edge
  timeout: 5m0s
status:
  conditions:
  - lastTransitionTime: "2022-09-07T16:12:40Z"
    message: reconciliation in progress
    reason: Progressing
    status: Unknown
    type: Ready
  inventory:
    entries:
    - id: _monarch-cache__Namespace
      v: v1
    - id: monarch-cache_monarch-couchdb__ServiceAccount
      v: v1
    - id: monarch-cache_monarch-couchdb_rbac.authorization.k8s.io_Role
      v: v1
    - id: monarch-cache_monarch-couchdb_rbac.authorization.k8s.io_RoleBinding
      v: v1
    - id: monarch-cache_monarch-cache-logger-properties__ConfigMap
      v: v1
    - id: monarch-cache_monarch-couchdb__ConfigMap
      v: v1
    - id: monarch-cache_monarch-file-snapshot-cron__ConfigMap
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-app-param__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-item__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-promotion__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-sis__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-sis-source__Secret
      v: v1
    - id: monarch-cache_monarch-cache-ops-secret-tax__Secret
      v: v1
    - id: monarch-cache_monarch-couchdb__Secret
      v: v1
    - id: monarch-cache_monarch-cache-admin-health-svc__Service
      v: v1
    - id: monarch-cache_monarch-couchdb__Service
      v: v1
    - id: monarch-cache_monarch-couchdb-metrics__Service
      v: v1
    - id: monarch-cache_monarch-couchdb-st1531__Service
      v: v1
    - id: monarch-cache_monarch-healthcheck-svc__Service
      v: v1
    - id: monarch-cache_monarch-svc-couchdb__Service
      v: v1
    - id: monarch-cache_monarch-cache-admin_apps_Deployment
      v: v1
    - id: monarch-cache_monarch-couchdb-metrics_apps_Deployment
      v: v1
    - id: monarch-cache_monarch-healthcheck_apps_Deployment
      v: v1
    - id: monarch-cache_monarch-couchdb_apps_StatefulSet
      v: v1
    - id: monarch-cache_monarch-cache-admin-pvc__PersistentVolumeClaim
      v: v1
    - id: monarch-cache_monarch-cache-couchdb-rules_monitoring.coreos.com_PrometheusRule
      v: v1
    - id: monarch-cache_monarch-couchdb-metrics_monitoring.coreos.com_ServiceMonitor
      v: v1
    - id: monarch-cache_monarch-couch-health_traefik.containo.us_IngressRoute
      v: v1alpha1
    - id: monarch-cache_monarch-couch-http_traefik.containo.us_IngressRoute
      v: v1alpha1
    - id: monarch-cache_monarch-file-fetch-http_traefik.containo.us_IngressRoute
      v: v1alpha1
  lastAppliedRevision: main/980a6714ecb85eb78623b124a1e98edc0eecc
  lastAttemptedRevision: main/980a6714ecb85eb78623b124a1e98edc0eecc
  observedGeneration: 2

[texasbobs]

One pod in the target namespace was not running. I deleted that pod and almost immediately the kustomization worked. Any thoughts on why it got hung and nothing was logged about it?

[stefanprodan]

Was it one of the satefulset pods that was failing to start? My guess it that it would error out after the timeout expires, have you waited 5 minutes and it was still stuck in progressing? Also are you running the latest Flux?

[texasbobs]

It was a deployment. The timeout and retryInterval were added by me but that was on the second day of it being stuck. We're running fluxcd/kustomize-controller:v0.22.3

[kingdonb]

Has this issue recurred since you saw it last week? If you are able to reproduce it consistently, I'll be interested to know what went wrong here, (even if it turns out to not have been a Flux issue)

[texasbobs]

That was the first and only time I've ever seen it on any of the 2100+ clusters we have with 27 kustomizations on each.