Client closed connection when having kafka-proxy exposed as a service

[JM322]

When deployed as sidecar container the kafka-proxy works as expected:

spec:
      containers:
      - name: kafka-proxy
        image: grepplabs/kafka-proxy:latest
        args:
        - server
        - --bootstrap-server-mapping=xxxxxxx.aws.confluent.cloud:9092,0.0.0.0:9092
        - --tls-enable
        - --forward-proxy=http://xxxxxxx:9400
        - --debug-enable
        - --log-level=debug
        - --log-format=json
      - name: kafkacat
        image: confluentinc/cp-kafkacat
       ...

Since I don't want to have a sidecar container for all of my services I am trying to set up the kafka-proxy as a service that can be used by all applications:

apiVersion: v1
kind: Service
metadata:
  name: kafka-proxy
  namespace: kafka-proxy
  labels:
    app: kafka-proxy
spec:
  ports:
  - port: 9092
    targetPort: 9092
    name: kafka
  clusterIP: None
  selector:
    app: kafka-proxy
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-proxy
  namespace: kafka-proxy
spec:
  selector:
    matchLabels:
      app: kafka-proxy
  replicas: 1
  serviceName: kafka-proxy
  template:
    metadata:
      labels:
        app: kafka-proxy
    spec:
      containers:
      - name: kafka-proxy
        image: grepplabs/kafka-proxy:latest
        args:
        - 'server'
        - '--log-format=json'
        - "--log-level=debug"
        - '--bootstrap-server-mapping=xxxxxxxx.confluent.cloud:9092,0.0.0.0:9092,kafka-proxy.kafka-proxy.svc.cluster.local:9092'
        - '--forward-proxy=http://xxxxxxx:9400'
        - '--tls-enable'
        ports:
        - name: metrics
          containerPort: 9080
        - name: confluent-kafka
          containerPort: 9092
        livenessProbe:
          httpGet:
            path: /health
            port: 9080
          initialDelaySeconds: 5
          periodSeconds: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 9080
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 2
          failureThreshold: 5
      restartPolicy: Always

When I now try to reach the Kafka I get a ERROR: Failed to query metadata for topic TOPIC-NAME: Local: Broker transport failure in the client and the following logs in the kafka-proxy:

{"@level":"info","@message":"Starting kafka-proxy version v0.3.12","@timestamp":"2024-11-26T10:19:36Z"}
{"@level":"info","@message":"Bootstrap server xxxxxxxxx.confluent.cloud:9092 advertised as kafka-proxy.kafka-proxy.svc.cluster.local:9092","@timestamp":"2024-11-26T10:19:36Z"}    
{"@level":"info","@message":"Listening on 0.0.0.0:9092 ([::]:9092) for remote xxxxxxxxx.confluent.cloud:9092","@timestamp":"2024-11-26T10:19:36Z"}
{"@level":"info","@message":"Kafka clients will connect through the HTTP proxy xxxxxxxxx:9400 using CONNECT","@timestamp":"2024-11-26T10:19:36Z"}
{"@level":"info","@message":"Ready for new connections","@timestamp":"2024-11-26T10:19:36Z"}
{"@level":"info","@message":"New connection for xxxxxxxxx.confluent.cloud:9092","@timestamp":"2024-11-26T10:20:03Z"}
{"@level":"debug","@message":"Kafka request key 18, version 0, length 21","@timestamp":"2024-11-26T10:20:04Z"}
{"@level":"info","@message":"Client closed local connection on 192.168.14.127:9092 from 192.168.190.31:36766 (xxxxxxxxx.confluent.cloud:9092)","@timestamp":"2024-11-26T10:20:08Z"}

Is there anything I missed with the configuration?