failed (add): failed to create macvlan: cannot assign requested address

[dmfrey]

I'm not able to assign addresses to my pod.

Multus is deployed with FluxCD, here is the HelmRelease:
https://github.com/dmfrey/home-gitops/blob/main/kubernetes/homelab-k8s-001/apps/network/multus/app/helmrelease.yaml

And my networks are configured with the following NetworkAttachmentDefinition:
https://github.com/dmfrey/home-gitops/blob/main/kubernetes/homelab-k8s-001/apps/network/multus/config/networkattachment-iot.yaml

My Talos cluster exposes the eth0.50 vlan

92.168.30.31   network     LinkStatus   eth0              13        ether               38:f7:cd:c6:11:4a                                 up           true
192.168.30.31   network     LinkStatus   eth0.30           12        ether      vlan     38:f7:cd:c6:11:4a                                 up           true
192.168.30.31   network     LinkStatus   eth0.50           11        ether      vlan     38:f7:cd:c6:11:4a                                 up           true

I'm applying a pod annotation to the pod in the HelmRelease:

pod:
  annotations:
     k8s.v1.cni.cncf.io/networks: |
      [{
        "name": "iot-vlan",
        "namespace": "network",
        "ips": ["192.168.50.8/24"],
        "mac": "f1:fd:59:0b:bd:2b"
      }]

When the pod starts its fire the following events.

Events:                                                                                                                                                                                                                                                                                            
  Type     Reason                  Age   From               Message                                                                                                                                                                                                                                
  ----     ------                  ----  ----               -------                                                                                                                                                                                                                                
  Normal   Scheduled               5s    default-scheduler  Successfully assigned download/admin-c5999f66d-djg6t to dmf-amd-003                                                                                                                                                                    
  Normal   AddedInterface          5s    multus             Add eth0 [172.22.0.238/32] from cilium                                                                                                                                                                                                 
  Warning  FailedCreatePodSandBox  4s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a56b77988d8ac693fdbaad50a8b0d03e237a689d6df2b59678e00b713d207da4": plugin type="multus" name="multus-cni-network" failed (add): [do
wnload/admin-c5999f66d-djg6t/2ff097cc-9ee0-48b3-a0b2-4d60efb80ab4:iot-vlan]: error adding container to network "iot-vlan": plugin type="macvlan" failed (add): failed to create macvlan: cannot assign requested address                                                                           

Is there some way to get around this issue?

Environment:

• 5 Talos Linux kubernetes nodes
• 3 control plane nodes
• 2 worker nodes
• Ubiquiti UDM Special Edition
• .30 vlan homelab (where the nodes live)
• .50 vlan iot

If there is anything else I can provide, please let me know. Thanks for any assistance.

[dougbtv]

Hey Daniel!

Ummm, so... alright, another reference config, here's the example I use for testing setting a static ip/mac at run time (e.g. via annotation)

https://gist.github.com/dougbtv/a1f2b40bb3e57429ab0dcd8ad4d96c42

I think you might need to add tuning into the CNI chain, my example:

"plugins": [{
          "type": "bridge",
          "bridge": "cnitest0",
          "name": "testbridge",
          "ipam": {
            "type": "static"
          }
        }, {
          "capabilities": { "mac": true, "ips": true },
          "type": "tuning"
        }]

and then your config...

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: iot-vlan
spec:
  config: |-
    {
    "cniVersion": "0.3.1",
    "name": "iot-vlan",
    "plugins": [
        {
            "type": "macvlan",
            "master": "eth0.50",
            "mode": "bridge",
            "ipam": {
                "type": "static",
                "routes": [
                    {
                        "dst": "192.168.50.0/24",
                        "gw": "192.168.50.1"
                    }
                ]
            }
        },
        {
            "capabilities": {
                "mac": true,
                "ips": true
            },
            "type": "tuning"
        }
    ]
}

Kinda just wildly pasted in an run through a json lint. I think that might be the piece that's missing is the tuning plugin.

Worth a shot at least.

[dougbtv]

Cool homelab setup, like, it's rad that it looks like you're doing some kind of isolation to handle the IoT network or something? So like you isolate that traffic on the pod to the IoT segment or something? Just curious what you're doing if you don't mind sharing, mostly I think it's neat.

[dmfrey]

@dougbtv Thanks!

I mostly am following the k8s-at-home generated projects.

There are 5 nodes in my cluster: 3 control-plane and 2 workers.

My main use case is this. I want to expose Home Assistant running in my homelab vlan (30) to my iot vlan (50) so that it can discover the devices and services running over there. This used to work, but for some reason, the last couple of months, it stopped and I was getting that message that it couldn't assign addresses. I have a test pod I've been playing with, but just sits there in a pending state with the message I posted.

There are other services I'd like to expose to that vlan as well, like ESPHome, etc. as well

Is there anything specific you'd like to know about it?

[dmfrey]

I updated the network definition, but I'm still getting the same error:

Events:                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  Type     Reason                  Age   From               Message                                                                                                                                                                                                                                                                                                                                                                                               
  ----     ------                  ----  ----               -------                                                                                                                                                                                                                                                                                                                                                                                               
  Normal   Scheduled               10s   default-scheduler  Successfully assigned download/admin-c5999f66d-vlvxv to dmf-amd-001                                                                                                                                                                                                                                                                                                                                   
  Normal   AddedInterface          10s   multus             Add eth0 [172.22.2.245/32] from cilium                                                                                                                                                                                                                                                                                                                                                                
  Warning  FailedCreatePodSandBox  10s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ad7bd8435ba9a9631a360a4f73254375f055f86a2df69eb58d22ce59542f2fd3": plugin type="multus" name="multus-cni-network" failed (add): [download/admin-c5999f66d-vlvxv/9bd3eb7e-0713-4998-8102-1e6419018123:iot-vlan]: error adding container to network "iot-vlan": plugin type="macvlan" failed (add): 
failed to create macvlan: cannot assign requested address                                                                                                                                                                                                                                                                                                                                                                                                         

Is there something potentially with cilium or anything I need to consider on the router (Mine is a Unifi Dream Machine Special Edition)?