[kube-state-metrics] make livenessProbe and readinessProbe optional

[r0bj]

What this PR does / why we need it

This attempt addresses the issue of probes failing, which occurred after the changes in PR #4982. By making the livenessProbe and readinessProbe optional, users can disable them when using kube-rbac-proxy.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes # [kube-state-metrics] Readiness probe not working with rbac proxy #4992

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[SuperQ]

This is probably not a good idea. Just disabling probes is going to have unintended consequences. Things like PDBs will no longer function.

It would be better to solve why probes are failing rather than just ignore them.

[r0bj]

@SuperQ Probes are failing because the port is bound to 127.0.0.1, so the kubelet cannot reach it. There are three potential solutions, each with its trade-offs:

1. Bind both main and telemetry ports to 0.0.0.0.
   In this case, the kubelet can access the ports via probes, but doing so defeats the purpose of using kube-rbac-proxy, which is intended to prevent unauthorized access to the kube-state-metrics endpoints.

2. Bind both main and telemetry ports to 127.0.0.1.
   This configuration hides the kube-state-metrics endpoints, allowing access only through the kube-rbac-proxy-protected ports. However, the kubelet will be unable to reach the kube-state-metrics ports, causing probes to fail.

3. Bind both main and telemetry ports to 127.0.0.1 and use the host network.
   With this approach, the kubelet can reach the kube-state-metrics endpoints, so probes work correctly, and the endpoints remain protected. The downside is that running pods with the host network may not be allowed due to namespace security restrictions (for example, the baseline and restricted pod security standards as documented in the Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/).

Given these options, perhaps we should provide a configuration option that allows users to choose their preferred trade-off based on their environment and security requirements. By configuration options, I mean a way to disable and enable probes independently.

[fahedouch]

@r0bj
Additional option:
Expose an endpoint on the kube-rbac-proxy to check the readiness/liveness of kube-state-metrics. This makes a lot of sense to me as the proxy is the only way to health check the kube-state-metrics port.

[r0bj]

Additional option:
Expose an endpoint on the kube-rbac-proxy to check the readiness/liveness of kube-state-metrics. This makes a lot of sense to me as the proxy is the only way to health check the kube-state-metrics port.

Yes, this option seems reasonable. I’ve just checked, and kube-rbac-proxy has a parameter that may be helpful for this purpose:

--ignore-paths strings                        Comma-separated list of paths against which kube-rbac-proxy pattern-matches the incoming request. If the requst matches, it will proxy the request without performing an authentication or authorization check. Cannot be used with --allow-paths.

[r0bj]

This is my proposal for implementing the option mentioned by @fahedouch — to use the ports exposed by kube-rbac-proxy for probes:: #5234

[dotdc]

Closing in favor of #5234