alert-provider cannot receive notifications

[ForcemCS]

I am trying to use Notification Controller and I have created the following resource,But when I update the code in the deployment repository (e.g. change the image version), I don't get any message from my Wechat, how can I troubleshoot this issue

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: weixin-provider
  namespace: flux-system
spec:
  type: generic
  address: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=2028207c-300e-4154-bfacxxxxxxxxxxxxxxxxxxxxx
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: weixin-notification
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: '*'
  - kind: GitRepository
    name: '*'
  - kind: HelmRepository
    name: '*'
  providerRef:
    name: weixin-provider

[stefanprodan]

Looking at Kubernetes events for that Alert with kubectl describe, or look at the notification-controller logs that should tell you if the endpoint returns an error. Also please post here the flux check output.

[ForcemCS]

#kubectl    -n flux-system  describe  alerts.notification.toolkit.fluxcd.io  weixin-notification  
Name:         weixin-notification
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>
API Version:  notification.toolkit.fluxcd.io/v1beta3
Kind:         Alert
Metadata:
  Creation Timestamp:  2024-02-21T10:49:14Z
  Generation:          1
  Resource Version:    33017022
  UID:                 d9f17d19-779a-408f-9e9e-2d58e9907d9f
Spec:
  Event Severity:  info
  Event Sources:
    Kind:  Kustomization
    Name:  *
    Kind:  GitRepository
    Name:  *
    Kind:  HelmRepository
    Name:  *
  Provider Ref:
    Name:  weixin-provider
Events:    <none>

I modified the deployment repository image=v3 and traced the notification-controller logs

{"level":"info","ts":"2024-02-22T09:30:17.839Z","logger":"receiver-server","msg":"handling request: f1bccd97ca9bc16064d941eb404a908a95575a101ee822a4991c484d62cae58d"}
{"level":"info","ts":"2024-02-22T09:30:17.842Z","logger":"receiver-server","msg":"handling GitLab event: Push Hook","reconciler kind":"Receiver","name":"gitlab-receiver-app-demo","namespace":"flux-system"}
{"level":"info","ts":"2024-02-22T09:30:17.850Z","logger":"receiver-server","msg":"resource 'GitRepository/app-demo.flux-system' annotated","reconciler kind":"Receiver","name":"gitlab-receiver-app-demo","namespace":"flux-system"}
{"level":"info","ts":"2024-02-22T09:30:18.298Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"GitRepository","namespace":"flux-system","name":"app-demo","uid":"3e5ae5f1-d78d-4fa6-9468-456c51290644","apiVersion":"source.toolkit.fluxcd.io/v1","resourceVersion":"33854193"},"message":"stored artifact for commit 'Update deployment.yaml'"}
{"level":"info","ts":"2024-02-22T09:30:18.319Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"vote-production","uid":"1d138a72-2b73-4b18-9e6c-23504c66e225","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33850835"},"message":"Dependencies do not meet ready condition, retrying in 30s"}
{"level":"info","ts":"2024-02-22T09:30:18.549Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"redis-production","uid":"9002be0f-939a-424f-970f-2300266960e6","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33851179"},"message":"Reconciliation finished in 230.799102ms, next run in 30m0s"}
{"level":"info","ts":"2024-02-22T09:30:48.432Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"vote-production","uid":"1d138a72-2b73-4b18-9e6c-23504c66e225","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33854203"},"message":"Deployment/instavote/vote configured"}
{"level":"info","ts":"2024-02-22T09:30:53.484Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"vote-production","uid":"1d138a72-2b73-4b18-9e6c-23504c66e225","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33854203"},"message":"Health check passed in 5.043091222s"}
{"level":"info","ts":"2024-02-22T09:30:53.521Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"vote-production","uid":"1d138a72-2b73-4b18-9e6c-23504c66e225","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33854203"},"message":"Reconciliation finished in 5.175641618s, next run in 1h0m0s"}
{"level":"info","ts":"2024-02-22T09:31:41.229Z","msg":"Reconciliation finished, next run in 10m0s","controller":"receiver","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Receiver","Receiver":{"name":"gitlab-receiver-app-demo","namespace":"flux-system"},"namespace":"flux-system","name":"gitlab-receiver-app-demo","reconcileID":"f7c2e1e9-26f5-4cba-8d95-47ed3c90f653"}
{"level":"debug","ts":"2024-02-22T09:31:41.229Z","logger":"events","msg":"Reconciliation finished, next run in 10m0s","type":"Normal","object":{"kind":"Receiver","namespace":"flux-system","name":"gitlab-receiver-app-demo","uid":"d6e3b99c-1525-467f-b0ec-29f01a6b63f7","apiVersion":"notification.toolkit.fluxcd.io/v1","resourceVersion":"33842861"},"reason":"Succeeded"}
{"level":"info","ts":"2024-02-22T09:32:16.738Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"flux-system","uid":"17e3fca4-96c6-434a-a9dd-a1e36a3cf4a3","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"33849289"},"message":"Service/flux-system/webhook-receiver dry-run failed (Invalid): Service \"webhook-receiver\" is invalid: spec.ports[1].name: Duplicate value: \"http\"\n"}

[stefanprodan]

The weixin webhook should've received events as shown in logs, if it didn't, then the problem is on the weixin side.

[ForcemCS]

root@helm:~/hw/server#  flux check  
► checking prerequisites
✗ flux 2.2.2 <2.2.3 (new CLI version is available, please upgrade)
✔ Kubernetes 1.27.3-r0-27.2.28 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.2
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.3
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.3
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

[ForcemCS]

I tested it with the following command and it was fine

curl 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=2028xxxxxx' \
   -H 'Content-Type: application/json' \
   -d '
   {
        "msgtype": "text",
        "text": {
            "content": "Hello Fluxcd "
        }
   }'

[stefanprodan]

The Flux events don't have that json fields.

[stefanprodan]

See here how the json looks: https://fluxcd.io/flux/components/notification/events/#example