[BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode #4952

MuNeNICK · 2025-01-30T01:43:34Z

Kube-OVN Version

v1.13.2

Cilium Version

v1.16.6

Kubernetes Version

v1.32.1

Operation-system/Kernel Version

"Ubuntu 22.04.5 LTS"
5.15.0-125-generic__

Description

Hello.

I am using Cilium Chain.

In this environment, when deploying multiple VPC NAT Gateways with the same lanIp, I'm encountering a bug where Cilium Endpoints overlap, preventing the creation of VPC NAT Gateways.
Is there any way to work around this?

While the first deployed vpc-nat-gw-vpc1-natgw1-0 is functioning normally, the second deployed vpc-nat-gw-vpc2-natgw1-0 is not working.

ubuntu@ubuntu:~$ kubectl get pod -n kube-system 
NAME                                   READY   STATUS              RESTARTS      AGE
cilium-envoy-9jpg5                     1/1     Running             0             29m
cilium-operator-f99b55df8-4lwsx        1/1     Running             0             29m
cilium-sgxqq                           1/1     Running             0             29m
coredns-668d6bf9bc-ljgq8               1/1     Running             0             25m
coredns-668d6bf9bc-qbpkq               1/1     Running             0             25m
etcd-ubuntu                            1/1     Running             0             33m
hubble-relay-59b94b6665-fjpdx          1/1     Running             0             29m
hubble-ui-69d69b64cf-j66sc             2/2     Running             0             29m
kube-apiserver-ubuntu                  1/1     Running             0             33m
kube-controller-manager-ubuntu         1/1     Running             0             33m
kube-multus-ds-zrxvp                   1/1     Running             0             29m
kube-ovn-cni-sgfgs                     1/1     Running             1 (27m ago)   29m
kube-ovn-controller-647586899f-s6pv6   1/1     Running             2 (18m ago)   29m
kube-ovn-monitor-66896f8657-wlfp2      1/1     Running             0             29m
kube-ovn-pinger-hnt5x                  1/1     Running             0             29m
kube-proxy-v4k64                       1/1     Running             0             32m
kube-scheduler-ubuntu                  1/1     Running             0             33m
ovn-central-d6f9479bf-j7rz8            1/1     Running             0             29m
ovs-ovn-9xb4x                          1/1     Running             0             29m
vpc-nat-gw-vpc1-natgw1-0               1/1     Running             0             22m
vpc-nat-gw-vpc2-natgw1-0               0/1     ContainerCreating   0             17m
ubuntu@ubuntu:~$

vpc-nat-gw1 yaml (working)

kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1-natgw1
spec:
  vpc: vpc1
  subnet: vpc1-subnet1
  lanIp: 10.0.1.2
  externalSubnets:
    - ovn-vpc-external-network

vpc-nat-gw2 yaml (failed)

kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: vpc2-natgw1
spec:
  vpc: vpc2
  subnet: vpc2-subnet1
  lanIp: 10.0.1.2
  externalSubnets:
    - ovn-vpc-external-network

vpc-nat-gw2 detail

ubuntu@ubuntu:~/vpc2$ kubectl describe pod -n kube-system vpc-nat-gw-vpc2-natgw1-0
Name:             vpc-nat-gw-vpc2-natgw1-0
Namespace:        kube-system
Priority:         0
Service Account:  default
Node:             ubuntu/192.168.0.10
Start Time:       Thu, 30 Jan 2025 01:21:31 +0000
Labels:           app=vpc-nat-gw-vpc2-natgw1
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=vpc-nat-gw-vpc2-natgw1-67d766dbb4
                  ovn.kubernetes.io/vpc-nat-gw=true
                  statefulset.kubernetes.io/pod-name=vpc-nat-gw-vpc2-natgw1-0
Annotations:      k8s.v1.cni.cncf.io/networks: kube-system/ovn-vpc-external-network
                  ovn-vpc-external-network.kube-system.kubernetes.io/allocated: true
                  ovn-vpc-external-network.kube-system.kubernetes.io/cidr: 192.168.0.0/24
                  ovn-vpc-external-network.kube-system.kubernetes.io/gateway: 192.168.0.1
                  ovn-vpc-external-network.kube-system.kubernetes.io/ip_address: 192.168.0.62
                  ovn-vpc-external-network.kube-system.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"192.168.0.1"}]
                  ovn.kubernetes.io/allocated: true
                  ovn.kubernetes.io/cidr: 10.0.1.0/24
                  ovn.kubernetes.io/gateway: 10.0.1.1
                  ovn.kubernetes.io/ip_address: 10.0.1.2
                  ovn.kubernetes.io/logical_router: vpc2
                  ovn.kubernetes.io/logical_switch: vpc2-subnet1
                  ovn.kubernetes.io/mac_address: d2:37:d1:61:a0:73
                  ovn.kubernetes.io/pod_nic_type: veth-pair
                  ovn.kubernetes.io/routed: true
                  ovn.kubernetes.io/routes: [{"dst":"10.96.0.0/12","gw":"10.0.1.1"}]
                  ovn.kubernetes.io/vpc_nat_gw: vpc2-natgw1
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    StatefulSet/vpc-nat-gw-vpc2-natgw1
Containers:
  vpc-nat-gw:
    Container ID:  
    Image:         docker.io/kubeovn/vpc-nat-gateway:v1.13.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      infinity
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2t672 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-2t672:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               8m38s                  default-scheduler  Successfully assigned kube-system/vpc-nat-gw-vpc2-natgw1-0 to ubuntu
  Warning  FailedCreatePodSandBox  8m36s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "db1b2ed1fafb7d936acac6319b0eaf2b1da70c3423e2622646a066a877d71165": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/vpc-nat-gw-vpc2-natgw1-0/38b125aa-d606-4266-9e76-cdb38e942ba7:generic-veth]: error adding container to network "generic-veth": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:10.0.1.2 is already in use"
  Warning  FailedCreatePodSandBox  8m34s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "63ed6141fa5d3cf43460a0ab3935fe6a8635833b8d72e60ad7de4e6f5fe25b48": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/vpc-nat-gw-vpc2-natgw1-0/38b125aa-d606-4266-9e76-cdb38e942ba7:generic-veth]: error adding container to network "generic-veth": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:10.0.1.2 is already in use"

kube-ovn controller logs

I0130 01:21:31.476470       7 vpc_nat_gateway.go:180] handle add/update vpc nat gateway vpc2-natgw1
I0130 01:21:31.624762       7 vpc_nat_gateway.go:180] handle add/update vpc nat gateway vpc2-natgw1
I0130 01:21:31.683143       7 pod.go:227] enqueue add pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:31.683160       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:31.825620       7 pod.go:507] sync pod kube-system/vpc-nat-gw-vpc2-natgw1-0 allocated
I0130 01:21:31.825647       7 ipam.go:61] allocate v4 192.168.0.62, v6 , mac  for kube-system/vpc-nat-gw-vpc2-natgw1-0 from subnet ovn-vpc-external-network
I0130 01:21:31.905537       7 ipam.go:73] allocating static ip 192.168.0.62 from subnet ovn-vpc-external-network
I0130 01:21:31.905559       7 ipam.go:103] allocate v4 192.168.0.62, mac  for kube-system/vpc-nat-gw-vpc2-natgw1-0 from subnet ovn-vpc-external-network
I0130 01:21:32.004370       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:32.017718       7 ipam.go:73] allocating static ip 10.0.1.2 from subnet vpc2-subnet1
I0130 01:21:32.017736       7 ipam.go:103] allocate v4 10.0.1.2, mac d2:37:d1:61:a0:73 for kube-system/vpc-nat-gw-vpc2-natgw1-0 from subnet vpc2-subnet1
I0130 01:21:32.019809       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:32.893212       7 pod.go:695] sync pod kube-system/vpc-nat-gw-vpc2-natgw1-0 routed
I0130 01:21:32.893558       7 vpc_nat_gateway.go:290] handle init vpc nat gateway vpc2-natgw1
I0130 01:21:33.132158       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:33.132176       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:33.134667       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
I0130 01:21:33.134685       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc2-natgw1-0
E0130 01:21:37.894106       7 vpc_nat_gateway.go:297] failed to get nat gw vpc2-natgw1 pod: pod is not active now
E0130 01:21:37.894174       7 controller.go:1315] "Unhandled Error" err="error syncing init vpc nat gateway \"vpc2-natgw1\": failed to get nat gw vpc2-natgw1 pod: pod is not active now, requeuing" logger="UnhandledError"
I0130 01:21:38.894191       7 vpc_nat_gateway.go:290] handle init vpc nat gateway vpc2-natgw1
E0130 01:21:43.894731       7 vpc_nat_gateway.go:297] failed to get nat gw vpc2-natgw1 pod: pod is not active now

cilium agent log

time="2025-01-30T01:21:33Z" level=info msg="Create endpoint request" addressing="&{10.0.1.2   fe80::d037:d1ff:fe61:a073  }" containerID=db1b2ed1fafb7d936acac6319b0eaf2b1da70c3423e2622646a066a877d71165 containerInterface=eth0 datapathConfiguration="&{false true false true true 0xc000c0db0c}" interface=db1b2ed1fafb_h k8sPodName=kube-system/vpc-nat-gw-vpc2-natgw1-0 k8sUID=38b125aa-d606-4266-9e76-cdb38e942ba7 labels="[]" subsys=daemon sync-build=true
time="2025-01-30T01:21:33Z" level=warning msg="Creation of endpoint failed due to invalid data" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=0 error="IP ipv4:10.0.1.2 is already in use" ipv4= ipv6= k8sPodName=/ subsys=daemon
time="2025-01-30T01:21:35Z" level=info msg="Create endpoint request" addressing="&{10.0.1.2   fe80::d037:d1ff:fe61:a073  }" containerID=63ed6141fa5d3cf43460a0ab3935fe6a8635833b8d72e60ad7de4e6f5fe25b48 containerInterface=eth0 datapathConfiguration="&{false true false true true 0xc0012f9d1c}" interface=63ed6141fa5d_h k8sPodName=kube-system/vpc-nat-gw-vpc2-natgw1-0 k8sUID=38b125aa-d606-4266-9e76-cdb38e942ba7 labels="[]" subsys=daemon sync-build=true
time="2025-01-30T01:21:35Z" level=warning msg="Creation of endpoint failed due to invalid data" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=0 error="IP ipv4:10.0.1.2 is already in use" ipv4= ipv6= k8sPodName=/ subsys=daemon

cilium endpoint list

ubuntu@ubuntu:~$ kubectl exec -it -n kube-system cilium-sgxqq -- cilium endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6                        IPv4        STATUS   
           ENFORCEMENT        ENFORCEMENT                                                                                                                                       
1016       Disabled           Disabled          3861       k8s:app=vpc-nat-gw-vpc1-natgw1                                               fe80::d825:60ff:febe:b50f   10.0.1.2    ready   
                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system                                                   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                     
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                                              
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                                                  
                                                           k8s:ovn.kubernetes.io/vpc-nat-gw=true                                                                                        
1309       Disabled           Disabled          27537      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fe80::8d4:3ff:fe5c:3658     10.16.0.7   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                     
                                                           k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                              
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                                                  
                                                           k8s:k8s-app=kube-dns                                                                                                         
1387       Disabled           Disabled          1          k8s:kube-ovn/role=master                                                                                             ready   
                                                           k8s:node-role.kubernetes.io/control-plane                                                                                    
                                                           k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                                  
                                                           k8s:ovn.kubernetes.io/ovs_dp_type=kernel                                                                                     
                                                           reserved:host                                                                                                                
2723       Disabled           Disabled          25790      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=vpc1          fe80::b089:ddff:fee0:f228   10.0.1.10   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                     
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                                              
                                                           k8s:io.kubernetes.pod.namespace=vpc1                                                                                         
3572       Disabled           Disabled          27537      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fe80::5049:38ff:fef9:c1cb   10.16.0.8   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                     
                                                           k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                              
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                                                  
                                                           k8s:k8s-app=kube-dns                                                                                                         
ubuntu@ubuntu:~$

Steps To Reproduce

environment setup:

# Install Kube-OVN
helm upgrade --install kube-ovn kubeovn/kube-ovn \
  --namespace kube-system \
  --set MASTER_NODES=${MASTER_IP} \
  --set func.ENABLE_NP=false \
  --set func.ENABLE_LB_SVC=true \
  --set func.ENABLE_TPROXY=true \
  --set cni_conf.CNI_CONFIG_PRIORITY=10

# Create CNI ConfigMap
cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cni-configuration
  namespace: kube-system
data:
  cni-config: |-
    {
      "name": "generic-veth",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "kube-ovn",
          "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
          "ipam": {
            "type": "kube-ovn",
            "server_socket": "/run/openvswitch/kube-ovn-daemon.sock"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "cilium-cni"
        }
      ]
    }
EOF

# Install Cilium
helm upgrade --install cilium cilium/cilium \
  --namespace kube-system \
  --set operator.replicas=1 \
  --set cni.chainingMode=generic-veth \
  --set cni.customConf=true \
  --set cni.configMap=cni-configuration \
  --set routingMode=native \
  --set enableIPv4Masquerade=false \
  --set devices="eth+ ovn0 genev_sys_6081 vxlan_sys_4789" \
  --set enableIdentityMark=false \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set daemon.enableSourceIPVerification=false

# Install Multus
curl -L https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml | \
kubectl apply -f -

# Create external network configuration
cat > temp.yaml << EOF
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: ovn-vpc-external-network
spec:
  protocol: IPv4
  provider: ovn-vpc-external-network.kube-system
  cidrBlock: 192.168.0.0/24
  gateway: 192.168.0.1
  excludeIps: 
      - 192.168.0.1..192.168.0.60
      - 192.168.0.100..192.168.0.254
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-vpc-external-network
  namespace: kube-system
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "kube-ovn",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "ovn-vpc-external-network.kube-system"
    }
  }'
EOF
kubectl apply -f temp.yaml

# vpc1 environment (working)
apiVersion: v1
kind: Namespace
metadata:
  name: vpc1
---
kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1
spec:
  staticRoutes:
    - cidr: 0.0.0.0/0
      nextHopIP: 10.0.1.2
      policy: policyDst
  namespaces:
  - vpc1
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1-subnet1
spec:
  vpc: vpc1
  cidrBlock: 10.0.1.0/24
  excludeIps:
    - 10.0.1.2
  protocol: IPv4
  namespaces:
    - vpc1
---
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1-natgw1
spec:
  vpc: vpc1
  subnet: vpc1-subnet1
  lanIp: 10.0.1.2
  externalSubnets:
    - ovn-vpc-external-network

# vpc2 environment (failed)
apiVersion: v1
kind: Namespace
metadata:
  name: vpc2
---
kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: vpc2
spec:
  staticRoutes:
    - cidr: 0.0.0.0/0
      nextHopIP: 10.0.1.2
      policy: policyDst
  namespaces:
  - vpc2
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: vpc2-subnet1
spec:
  vpc: vpc2
  cidrBlock: 10.0.1.0/24
  excludeIps:
    - 10.0.1.2
  protocol: IPv4
  namespaces:
    - vpc2
---
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: vpc2-natgw1
spec:
  vpc: vpc2
  subnet: vpc2-subnet1
  lanIp: 10.0.1.2
  externalSubnets:
    - ovn-vpc-external-network

Current Behavior

When trying to create VPC NAT Gateways with the same lanIp, they cannot be created properly.

Expected Behavior

VPC NAT Gateways should be created successfully even when they have the same lanIp.

The text was updated successfully, but these errors were encountered:

dosubot · 2025-01-30T01:45:46Z

I found a similar closed issue regarding the inability to use the same EIP for DNAT and SNAT in VPC gateways, which was resolved by a pull request that added support for shared EIP in iptables NAT ^[1].

In your case, the error arises because the same lanIp (10.0.1.2) is being used for multiple VPC NAT Gateways, which leads to IP conflicts in the Cilium chaining mode. Cilium reports that the IP is already in use, preventing the creation of the second VPC NAT Gateway.

To work around this issue, you should ensure that each VPC NAT Gateway has a unique lanIp. This will prevent IP conflicts and allow each gateway to be created successfully. If using the same lanIp is a requirement, you might need to explore network policies or configurations that allow IP sharing, but this typically involves more complex network setups and might not be supported directly by the current configuration.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

MuNeNICK · 2025-01-30T02:59:25Z

As a temporary solution, executing cilium endpoint disconnect {id} within the cilium agent Pod resolves the issue, but it's cumbersome to do this every time.

However, when deleting the Cilium endpoint, the existing VPC-NAT-GW stops functioning, and all except the latest one becomes unable to communicate.
This is a problem.

MuNeNICK · 2025-01-31T01:04:59Z

@oilbeater @zbb88888 @zhangzujian

Hi.

Is this within specifications?
I need to make a decision about whether to use Cilium integration.

I await your response.

zbb88888 · 2025-02-05T07:29:26Z

more than one VPC NAT Gateways with the same lanIp, the lanip are in different VPCs. I think it is a bug.

the pod can not be ready, cloud you please show the err log in the kube-ovn-cni. it will use arp or ping to check whether the gw is pingable.

zbb88888 · 2025-02-05T07:30:10Z

maybe something blocks the ping packet.

zhangzujian · 2025-02-06T03:08:52Z

It seems that Cilium does not support same/conflict endpoint IPs. I suggest you to report this problem to the Cilium community.

MuNeNICK · 2025-02-18T01:10:14Z

It seems that Cilium does not support same/conflict endpoint IPs. I suggest you to report this problem to the Cilium community.

@zhangzujian
I am currently creating an Issue in Cilium, but I expect this will be handled as a Kube-OVN issue (since having identical IPs within a cluster is not supposed to happen in the first place).
While Kube-OVN can work with Cilium, I believe this is an issue that many people will encounter.
Is there any intention on the Kube-OVN side to implement countermeasures for this?

zhangzujian · 2025-02-18T02:01:52Z

I expect this will be handled as a Kube-OVN issue (since having identical IPs within a cluster is not supposed to happen in the first place).

You are creating two vpc-nat-gateways for two different VPCs. VPCs are isolated from each other and different VPCs can have identical CIDRs/IPs. This is not a bug but a feature.

BTW, underlay subnets can also have identical CIDRs/IPs since they are working in a different netwoking mode.

MuNeNICK · 2025-02-18T02:22:39Z

I expect this will be handled as a Kube-OVN issue (since having identical IPs within a cluster is not supposed to happen in the first place).

You are creating two vpc-nat-gateways for two different VPCs. VPCs are isolated from each other and different VPCs can have identical CIDRs/IPs. This is not a bug but a feature.

BTW, underlay subnets can also have identical CIDRs/IPs since they are working in a different netwoking mode.

Thank you for your reply.

I understand that using VPC functionality allows for the use of identical IPs. However, I believe vpc-nat-gw might be an exception.
The vpc-nat-gw Pods are deployed under kube-system, and Pods in the kube-system namespace are deployed with the same IP address "10.0.1.2".
Furthermore, when Cilium integration is in place, Cilium's Endpoint assignment is executed before Kube-OVN's Endpoint assignment, resulting in an error when creating the second vpc-nat-gw Pod.

Is it a matter of CNI priority that Cilium performs Endpoint assignment before Kube-OVN?
I followed the official documentation to set up the Cilium integration environment, but would reviewing these steps help resolve the issue?
https://kubeovn.github.io/docs/v1.13.x/en/advance/with-cilium/

I plan to test by modifying the CNI priority.

MuNeNICK · 2025-02-18T03:12:24Z

As I predicted, when I increased Kube-OVN's priority by setting a lower configuration number than Cilium's and rebuilt the environment, vpc-nat-gw Pods with identical IPs were able to deploy normally without any Cilium Endpoint assignment errors.

ubuntu@ubuntu:~$ sudo ls /etc/cni/net.d/
00-multus.conf  01-kube-ovn.conflist  05-cilium.conflist  multus.d
ubuntu@ubuntu:~$

ubuntu@ubuntu:~$ kubectl get pod -A -o wide
NAMESPACE     NAME                                   READY   STATUS    RESTARTS        AGE     IP             NODE     NOMINATED NODE   READINESS GATES
kube-system   cilium-envoy-hkbvd                     1/1     Running   0               8m13s   192.168.0.20   ubuntu   <none>           <none>
kube-system   cilium-operator-78f5fdf98f-drhh6       1/1     Running   1 (6m26s ago)   8m13s   192.168.0.20   ubuntu   <none>           <none>
kube-system   cilium-tkftt                           1/1     Running   0               8m13s   192.168.0.20   ubuntu   <none>           <none>
kube-system   coredns-668d6bf9bc-ntb4n               1/1     Running   0               40s     10.16.0.19     ubuntu   <none>           <none>
kube-system   coredns-668d6bf9bc-qffsk               1/1     Running   0               25s     10.16.0.20     ubuntu   <none>           <none>
kube-system   etcd-ubuntu                            1/1     Running   0               12m     192.168.0.20   ubuntu   <none>           <none>
kube-system   hubble-relay-75d5bdf84b-mlpt4          1/1     Running   0               8m13s   10.16.0.5      ubuntu   <none>           <none>
kube-system   hubble-ui-69d69b64cf-vfbvt             2/2     Running   0               8m13s   10.16.0.6      ubuntu   <none>           <none>
kube-system   kube-apiserver-ubuntu                  1/1     Running   0               12m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-controller-manager-ubuntu         1/1     Running   2 (6m34s ago)   12m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-multus-ds-2txkf                   1/1     Running   0               4m58s   192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-ovn-cni-cs626                     1/1     Running   0               11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-ovn-controller-59c7b45555-svvgz   1/1     Running   1 (6m21s ago)   11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-ovn-monitor-7df749fb4-b4xl6       1/1     Running   0               11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-ovn-pinger-5hm4l                  1/1     Running   0               11m     10.16.0.2      ubuntu   <none>           <none>
kube-system   kube-proxy-nsvdv                       1/1     Running   0               11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   kube-scheduler-ubuntu                  1/1     Running   1 (6m27s ago)   12m     192.168.0.20   ubuntu   <none>           <none>
kube-system   ovn-central-5bbcc6b688-bx99t           1/1     Running   0               11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   ovs-ovn-ll4pf                          1/1     Running   0               11m     192.168.0.20   ubuntu   <none>           <none>
kube-system   vpc-nat-gw-vpc1-natgw1-0               1/1     Running   0               61s     10.0.1.254     ubuntu   <none>           <none>
kube-system   vpc-nat-gw-vpc2-natgw1-0               1/1     Running   0               9s      10.0.1.254     ubuntu   <none>           <none>
ubuntu@ubuntu:~$

Why does the official Cilium integration procedure set Cilium's priority higher than Kube-OVN?
https://kubeovn.github.io/docs/v1.13.x/en/advance/with-cilium/

zhangzujian · 2025-02-18T03:22:32Z

I think this is what happened:

User created two vpcs with identical subnet CIDR;
User created two vpc-nat-gateways with identical .spec.lanIp;
Kube-OVN created two vpc-nat-gateway statefulsets and then two pods were created automatically;
Multus-CNI (called by kubelet) handled the pod creation and called CNI plugins kube-ovn, portmap and cilium-cni sequentially;
Result of kube-ovn plugin was finally passed to cilium-cni and an error was reported/returned;
kubelet reported the error as shown in the pod event(s).

As I predicted, when I increased Kube-OVN's priority by setting a lower configuration number than Cilium's and rebuilt the environment, vpc-nat-gw Pods with identical IPs were able to deploy normally without any Cilium Endpoint assignment errors.

Cilium chaining is supposed to be disabled.

MuNeNICK · 2025-02-18T03:34:53Z

Cilium chaining is supposed to be disabled.

Ah, I apologize. Indeed, the Cilium chain was not functioning, and Hubble and Cilium Network Policy were not working.

I think this is what happened:

Yes, I believe that's exactly what was happening.
I would like Cilium to not throw errors under these conditions, but would this require a fix on Cilium's side rather than Kube-OVN's?

zhangzujian · 2025-02-18T03:42:10Z

would this require a fix on Cilium's side rather than Kube-OVN's?

Cilium, of course.

MuNeNICK · 2025-02-18T05:22:48Z

Cilium, of course.

I understand. I'll check if Cilium can address this issue.

If that's not possible, I'd like to explore alternative solutions.
My core requirement is that while users need the flexibility to freely specify VPCs and subnets, this isn't necessary for vpc-nat-gw.
For example, it would be acceptable to have a dedicated subnet for vpc-nat-gw (100.100.0.0/16) and assign lanIps from this range in a way that avoids overlap within the cluster.

To implement such a scenario, what configurations are possible with Kube-OVN?
Can we create usable subnets from both VPC1 and VPC2? Inter-VPC communication isn't necessary in this case.
I simply want to manage IP addresses across VPCs.

zhangzujian · 2025-02-18T05:50:39Z

If you are using Kube-OVN VPC and Cilium chaining, there is a chance that multiple normal pods are assigned identical IPs by Kube-OVN. This may also trigger cilium endpoint error.

zhangzujian · 2025-02-18T05:52:28Z

The root cause may be that Cilium does not support VPC or network isolation, so identical endpoints/IPs are invalid.

MuNeNICK · 2025-02-18T06:00:58Z

I see. So, in environments using multiple VPCs, it would be better not to use the Cilium chain?
I was considering Cilium Network Policy for implementing security group functionality, but I'll re-evaluate Kube-OVN's security group features (though I had some concerns about Kube-OVN's security groups).

zhangzujian · 2025-02-18T06:05:10Z

So, in environments using multiple VPCs, it would be better not to use the Cilium chain?

This is exactly what I mean.

MuNeNICK added the bug Something isn't working label Jan 30, 2025

dosubot bot added subnet vpc labels Jan 30, 2025

MuNeNICK changed the title ~~[BUG] Error when deploying multiple VPC-NAT-GWs with same IP in Cilium chain mode~~ [BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode Jan 30, 2025

MuNeNICK mentioned this issue Feb 6, 2025

Cilium endpoint allocation fails when identical IP addresses exist within the cluster. cilium/cilium#37459

Open

3 tasks

zhangzujian removed the bug Something isn't working label Feb 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode #4952

[BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode #4952

MuNeNICK commented Jan 30, 2025 •

edited

Loading

dosubot bot commented Jan 30, 2025

MuNeNICK commented Jan 30, 2025 •

edited

Loading

MuNeNICK commented Jan 31, 2025 •

edited

Loading

zbb88888 commented Feb 5, 2025

zbb88888 commented Feb 5, 2025

zhangzujian commented Feb 6, 2025

MuNeNICK commented Feb 18, 2025 •

edited

Loading

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025 •

edited

Loading

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

[BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode #4952

[BUG] Error when deploying multiple VPC-NAT-GWs with same lanIP in Cilium chain mode #4952

Comments

MuNeNICK commented Jan 30, 2025 • edited Loading

Kube-OVN Version

Cilium Version

Kubernetes Version

Operation-system/Kernel Version

Description

Steps To Reproduce

Current Behavior

Expected Behavior

dosubot bot commented Jan 30, 2025

MuNeNICK commented Jan 30, 2025 • edited Loading

MuNeNICK commented Jan 31, 2025 • edited Loading

zbb88888 commented Feb 5, 2025

zbb88888 commented Feb 5, 2025

zhangzujian commented Feb 6, 2025

MuNeNICK commented Feb 18, 2025 • edited Loading

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025 • edited Loading

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

MuNeNICK commented Feb 18, 2025

zhangzujian commented Feb 18, 2025

MuNeNICK commented Jan 30, 2025 •

edited

Loading

MuNeNICK commented Jan 30, 2025 •

edited

Loading

MuNeNICK commented Jan 31, 2025 •

edited

Loading

MuNeNICK commented Feb 18, 2025 •

edited

Loading

MuNeNICK commented Feb 18, 2025 •

edited

Loading