Trying to understand how to enable the breno1 after reboot.

[jasvinder1107]

Team i have installed the ovn-kubernetes via helm chart on k0s distribution(currently single controller). Everything works with installation. But after reboot my breno1 (default gateway bridge) doesn't get the IP at all. I am not understanding why because during installation eno1 (which is port with default gw) gets taken over by breno1. After manually plumbing the IP and default gateway for breno1 everything works. But i have to do this from console. Can you please help me if this is a bug in ovn-node ?

I have two port on the system.

1. eno1
2. eno2 (I wanted to use this port for Kubevirt Multi homing)

netplan is configured to have default gw on eno1. There is no known config for breno1 created after the OVN deployment in netplan by me.
My question is if this works first time and breno1 takes over eno1, why it fails to come up after the reboot. The CNI should handle it on reboot as well.

I0214 20:39:24.648018    5990 ovs.go:163] Exec(23): stderr: ""
I0214 20:39:24.648054    5990 ovs.go:159] Exec(24): /usr/bin/ovs-vsctl --timeout=15 get Port patch-breno1_bigbender-to-br-int Interfaces
I0214 20:39:24.652122    5990 ovs.go:162] Exec(24): stdout: "[f42769f5-c8ef-4fae-b17f-9f1e427b4acf]\n"
I0214 20:39:24.652163    5990 ovs.go:163] Exec(24): stderr: ""
I0214 20:39:24.652223    5990 ovs.go:159] Exec(25): /usr/bin/ovs-vsctl --timeout=15 get Interface a183cdb2-340f-4399-b35e-f97eaedafc8d Type
I0214 20:39:24.656237    5990 ovs.go:162] Exec(25): stdout: "\"\"\n"
I0214 20:39:24.656270    5990 ovs.go:163] Exec(25): stderr: ""
I0214 20:39:24.656299    5990 ovs.go:159] Exec(26): /usr/bin/ovs-vsctl --timeout=15 get Interface f42769f5-c8ef-4fae-b17f-9f1e427b4acf Type
I0214 20:39:24.660309    5990 ovs.go:162] Exec(26): stdout: "patch\n"
I0214 20:39:24.660347    5990 ovs.go:163] Exec(26): stderr: ""
I0214 20:39:24.660381    5990 ovs.go:159] Exec(27): /usr/bin/ovs-vsctl --timeout=15 br-get-external-id breno1 bridge-uplink
I0214 20:39:24.664572    5990 ovs.go:162] Exec(27): stdout: "eno1\n"
I0214 20:39:24.664604    5990 ovs.go:163] Exec(27): stderr: ""
I0214 20:39:24.664629    5990 ovs.go:159] Exec(28): /usr/bin/ovs-vsctl --timeout=15 --if-exists get interface breno1 mac_in_use
I0214 20:39:24.668637    5990 ovs.go:162] Exec(28): stdout: "\"00:26:xx:xx:xx"\n"
I0214 20:39:24.668676    5990 ovs.go:163] Exec(28): stderr: ""
I0214 20:39:24.668711    5990 ovs.go:159] Exec(29): /usr/sbin/sysctl -w net.ipv4.conf.breno1.forwarding=1
I0214 20:39:24.669665    5990 ovs.go:162] Exec(29): stdout: "net.ipv4.conf.breno1.forwarding = 1\n"
I0214 20:39:24.669703    5990 ovs.go:163] Exec(29): stderr: ""
I0214 20:39:24.669749    5990 ovs.go:159] Exec(30): /usr/bin/ovs-vsctl --timeout=15 --if-exists get Open_vSwitch . external_ids:ovn-bridge-mappings
I0214 20:39:24.673698    5990 ovs.go:162] Exec(30): stdout: "\"exgwphysnet:breno1,physnet:breno1\"\n"
I0214 20:39:24.673741    5990 ovs.go:163] Exec(30): stderr: ""
I0214 20:39:24.673790    5990 ovs.go:159] Exec(31): /usr/bin/ovs-vsctl --timeout=15 set Open_vSwitch . external_ids:ovn-bridge-mappings=exgwphysnet:breno1,physnet:breno1
I0214 20:39:24.677848    5990 ovs.go:162] Exec(31): stdout: ""
I0214 20:39:24.677903    5990 ovs.go:163] Exec(31): stderr: ""
I0214 20:39:24.677936    5990 ovs.go:159] Exec(32): /usr/bin/ovs-vsctl --timeout=15 port-to-br breno1
I0214 20:39:24.682049    5990 ovs.go:162] Exec(32): stdout: ""
I0214 20:39:24.682087    5990 ovs.go:163] Exec(32): stderr: "ovs-vsctl: no port named breno1\n"
I0214 20:39:24.682117    5990 ovs.go:165] Exec(32): err: exit status 1
I0214 20:39:24.682151    5990 ovs.go:159] Exec(33): /usr/bin/ovs-vsctl --timeout=15 br-exists breno1
I0214 20:39:24.686390    5990 ovs.go:162] Exec(33): stdout: ""
I0214 20:39:24.686432    5990 ovs.go:163] Exec(33): stderr: ""
I0214 20:39:24.686462    5990 ovs.go:159] Exec(34): /usr/bin/ovs-vsctl --timeout=15 list-ports breno1
I0214 20:39:24.692464    5990 ovs.go:162] Exec(34): stdout: "eno1\npatch-breno1_bigbender-to-br-int\n"
I0214 20:39:24.692509    5990 ovs.go:163] Exec(34): stderr: ""
I0214 20:39:24.692534    5990 ovs.go:159] Exec(35): /usr/bin/ovs-vsctl --timeout=15 get Port eno1 Interfaces
I0214 20:39:24.697155    5990 ovs.go:162] Exec(35): stdout: "[a183cdb2-340f-4399-b35e-f97eaedafc8d]\n"
I0214 20:39:24.697192    5990 ovs.go:163] Exec(35): stderr: ""
I0214 20:39:24.697225    5990 ovs.go:159] Exec(36): /usr/bin/ovs-vsctl --timeout=15 get Port patch-breno1_bigbender-to-br-int Interfaces
I0214 20:39:24.703108    5990 ovs.go:162] Exec(36): stdout: "[f42769f5-c8ef-4fae-b17f-9f1e427b4acf]\n"
I0214 20:39:24.703152    5990 ovs.go:163] Exec(36): stderr: ""
I0214 20:39:24.703189    5990 ovs.go:159] Exec(37): /usr/bin/ovs-vsctl --timeout=15 get Interface a183cdb2-340f-4399-b35e-f97eaedafc8d Type
I0214 20:39:24.709327    5990 ovs.go:162] Exec(37): stdout: "\"\"\n"
I0214 20:39:24.709355    5990 ovs.go:163] Exec(37): stderr: ""
I0214 20:39:24.709373    5990 ovs.go:159] Exec(38): /usr/bin/ovs-vsctl --timeout=15 get Interface f42769f5-c8ef-4fae-b17f-9f1e427b4acf Type
I0214 20:39:24.714685    5990 ovs.go:162] Exec(38): stdout: "patch\n"
I0214 20:39:24.714718    5990 ovs.go:163] Exec(38): stderr: ""
I0214 20:39:24.714746    5990 ovs.go:159] Exec(39): /usr/bin/ovs-vsctl --timeout=15 br-get-external-id breno1 bridge-uplink
I0214 20:39:24.720256    5990 ovs.go:162] Exec(39): stdout: "eno1\n"
I0214 20:39:24.720288    5990 ovs.go:163] Exec(39): stderr: ""
I0214 20:39:24.720314    5990 ovs.go:159] Exec(40): /usr/bin/ovs-vsctl --timeout=15 get interface eno1 ofport
I0214 20:39:24.726483    5990 ovs.go:162] Exec(40): stdout: "1\n"
I0214 20:39:24.726514    5990 ovs.go:163] Exec(40): stderr: ""
E0214 20:39:24.726758    5990 node_controller_manager.go:162] Stopping node network controller manager, err=failed to start default node network controller: bridge for interface failed for breno1: failed to get interface details for breno1: failed to find IPv4 address on interface breno1
I0214 20:39:24.726829    5990 ovnkube.go:587] Stopping ovnkube...
I0214 20:39:24.726840    5990 metrics.go:553] Stopping metrics server at address "192.168.10.10:9476"
I0214 20:39:24.728474    5990 reflector.go:311] Stopping reflector *v1.AdminPolicyBasedExternalRoute (0s) from pkg/crd/adminpolicybasedroute/v1/apis/informers/externalversions/factory.go:140
I0214 20:39:24.728820    5990 reflector.go:311] Stopping reflector *v1.EgressService (0s) from pkg/crd/egressservice/v1/apis/informers/externalversions/factory.go:140
I0214 20:39:24.729209    5990 reflector.go:311] Stopping reflector *v1.Namespace (0s) from k8s.io/client-go/informers/factory.go:160
I0214 20:39:24.733826    5990 reflector.go:311] Stopping reflector *v1.EgressIP (0s) from pkg/crd/egressip/v1/apis/informers/externalversions/factory.go:140
I0214 20:39:24.741491    5990 factory.go:656] Stopping watch factory
I0214 20:39:24.742548    5990 reflector.go:311] Stopping reflector *v1.EndpointSlice (0s) from k8s.io/client-go/informers/factory.go:160
I0214 20:39:24.742609    5990 reflector.go:311] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:160
I0214 20:39:24.742705    5990 reflector.go:311] Stopping reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:160
I0214 20:39:24.742938    5990 reflector.go:311] Stopping reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:160
I0214 20:39:24.743683    5990 ovnkube.go:591] Stopped ovnkube
I0214 20:39:24.743748    5990 metrics.go:553] Stopping metrics server at address "192.168.10.10:9410"
F0214 20:39:24.743923    5990 ovnkube.go:137] failed to run ovnkube: failed to start node network controller: failed to start default node network controller: bridge for interface failed for breno1: failed to get interface details for breno1: failed to find IPv4 address on interface breno1
info: Waiting for process_ready ovnkube to come up, waiting 5s ...
info: Waiting for process_ready ovnkube to come up, waiting 5s ...
error: process_ready ovnkube did not come up, exiting

Upon plumbing the ip on brdige and pushing default gw everything works after the reboot as well.

Can you help me with a way to solve this problem so i don't have to manually plumb the IP on breno1

my netplan looks like below.

root@bigbender:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eno1:
            dhcp4: false
            addresses : [192.168.10.10/24]
            nameservers:
              addresses: [192.168.10.1]
            routes:
            - to: default
              via: 192.168.10.1
        eno2:
            dhcp4: false
            addresses: [192.168.40.10/24]
    version: 2

root@bigbender:~# 

I can confirm the eno1 gets taken over by ovs-system after the reboot, but ip remains plumbed on eno1 , since that is only default GW. And breno1 doesn't take over the IP.

I have updated these values chart to deploy it .

root@bigbender:~/ovn-kubernetes/helm/ovn-kubernetes# git diff
diff --git a/helm/ovn-kubernetes/values-no-ic.yaml b/helm/ovn-kubernetes/values-no-ic.yaml
index 27fcb0093..9aa04c8a9 100644
--- a/helm/ovn-kubernetes/values-no-ic.yaml
+++ b/helm/ovn-kubernetes/values-no-ic.yaml
@@ -11,11 +11,11 @@ tags:
   ovnkube-zone-controller: false
 
 # -- Endpoint of Kubernetes api server
-k8sAPIServer: https://172.25.0.2:6443
+k8sAPIServer: https://localhost:6443
 # -- IP range for Kubernetes pods, /14 is the top level range, under which each /23 range will be assigned to a node
 podNetwork: 10.244.0.0/16/24
 # -- A comma-separated set of CIDR notation IP ranges from which k8s assigns service cluster IPs. This should be the same as the value provided for kube-apiserver "--service-cluster-ip-range" option
-serviceNetwork: 10.96.0.0/16
+serviceNetwork: 10.44.0.0/16
 # -- MTU of network interface in a Kubernetes pod
 mtu: 1400
 # -- Whether or not call `lookup` Helm function, set it to `true` if you want to run `helm dry-run/template/lint`
@@ -25,7 +25,7 @@ global:
   # -- The net device to be used for management port, will be renamed to ovn-k8s-mp0 and used to allow host network services and pods to access k8s pod and service networks
   nodeMgmtPortNetdev: ""
   # -- The interface on nodes that will be used for external gateway network traffic
-  extGatewayNetworkInterface: ""
+  extGatewayNetworkInterface: "eno1"
   # -- GENEVE UDP port (default 6081)
   encapPort: 6081
   # -- The gateway mode (shared or local), if not given, gateway functionality is disabled
root@bigbender:~/ovn-kubernetes/helm/ovn-kubernetes# 

Alpine pod is confirming that CNI is working after reboot and manual plumb of breno1

root@bigbender:~/ovn-kubernetes/helm/ovn-kubernetes# kubectl get pods  -A -o wide 
NAMESPACE        NAME                                READY   STATUS    RESTARTS        AGE    IP              NODE        NOMINATED NODE   READINESS GATES
default          alpine                              1/1     Running   0               110s   10.244.0.3      bigbender   <none>           <none>
kube-system      coredns-5555f45c94-qw5x7            1/1     Running   2 (8m5s ago)    81m    10.244.0.5      bigbender   <none>           <none>
kube-system      kube-proxy-k9kqz                    1/1     Running   2 (8m5s ago)    81m    192.168.10.10   bigbender   <none>           <none>
kube-system      metrics-server-7778865875-r2xc7     1/1     Running   2 (8m5s ago)    81m    10.244.0.4      bigbender   <none>           <none>
ovn-kubernetes   ovnkube-db-555b5f574b-dzw9d         2/2     Running   4 (8m5s ago)    80m    192.168.10.10   bigbender   <none>           <none>
ovn-kubernetes   ovnkube-identity-7f45b9d895-nrzfn   1/1     Running   2 (8m5s ago)    80m    192.168.10.10   bigbender   <none>           <none>
ovn-kubernetes   ovnkube-master-55f7587f6b-66xdl     2/2     Running   4 (8m5s ago)    80m    192.168.10.10   bigbender   <none>           <none>
ovn-kubernetes   ovnkube-node-l8rsj                  3/3     Running   6 (5m10s ago)   34m    192.168.10.10   bigbender   <none>           <none>
ovn-kubernetes   ovs-node-5tq9k                      1/1     Running   2 (8m5s ago)    80m    192.168.10.10   bigbender   <none>           <none>
root@bigbender:~/ovn-kubernetes/helm/ovn-kubernetes#