Upgrade from 0.11.0 to 0.12.0 failed

[shanduur]

What steps did you take and what happened:

Helm install failed for release capi-operator-system/capi-operator with chart [email protected]: cannot patch "addonproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "addonproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "bootstrapproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "bootstrapproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "controlplaneproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "controlplaneproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "coreproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "coreproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "infrastructureproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "infrastructureproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "ipamproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "ipamproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block && cannot patch "runtimeextensionproviders.operator.cluster.x-k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "runtimeextensionproviders.operator.cluster.x-k8s.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

What did you expect to happen:
Installation succeeded

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-operator version: 0.11.0 -> 0.12.0 -> 0.16.0
• Cluster-api version:
• Minikube/KIND version: N/A
• Kubernetes version: (use kubectl version): v1.31.5
• OS (e.g. from /etc/os-release): Debian

/kind bug
/area artifacts

[shanduur]

Upgrading to 0.16.0 still shows the same error. Removing release and installing 0.16.0 worked fine, but this looks like operational nightmare, as required me to remove all CRDs to complete the installation.

[furkatgofurov7]

/triage needs-information

Thanks for reporting the issue.

The problem causing this is that our CRDs have an invalid CA Bundle (starting from k8s v1.31 specifically) that needs to be removed (a more detailed explanation on this topic is in https://kubernetes.slack.com/archives/C0EG7JC6T/p1722441161968339 slack thread).

We removed the CA bundle in #591 but did not backport it to older releases (sorry for that), and it was part of the
https://github.com/kubernetes-sigs/cluster-api-operator/releases/tag/v0.14.0 release and onwards. So, if you had the operator release installed >=v0.14.0 from the beginning and upgraded to newer releases of the operator, you would not have seen it 😄 .

Since we can't backport the changes to the v0.11/2/3 release series anymore, simply because we don't support those older branches (currently we are maintaining the latest - 1 release series, v0.16 & v0.15), can we close this issue?