Autoscaled nodes not joining cluster

[enter-marlah]

Hello!

We are running Hetzner-k3s version 2.0.8 with the following worker pool config:

worker_node_pools:
- name: med-static
  instance_type: cpx31
  instance_count: 3
  location: hel1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 6

The nodes are created in Hetzner after autoscaling is initiated by stressing the cluster but they are not joining the cluster after that. We can ssh into the machines but they don't have for example ssh keys set or anything related to k3s installed. For static nodes the ssh keys are set correctly.

We think this has something to do with the previous cloud init wait problem in issue #379

If we read the code correctly the cloud_init_wait.sh script is not called when creating a autoscaled node?

We are running a private network only cluster. Regarding to this PR #458 our cloud init takes several minutes with both static and autoscaled nodes.

[vitobotta]

Hi, can you share your full config file (minus the token)?

[enter-marlah]

---
cluster_name: kube-prod
kubeconfig_path: "./kubeconfig"
k3s_version: v1.29.3+k3s1
include_instance_type_in_instance_name: true

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "./id_rsa_hetzner_prod.pub"
    private_key_path: "./id_rsa_hetzner_prod"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0
  public_network:
    ipv4: false
    ipv6: false
  private_network:
    enabled : true
    subnet: 10.0.0.0/16
    existing_network_name: "KubeNet"
  cni:
    enabled: true
    encryption: false
    mode: flannel

manifests:
  cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml"
  csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.8.0/deploy/kubernetes/hcloud-csi.yml"
  system_upgrade_controller_deployment_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/system-upgrade-controller.yaml"
  system_upgrade_controller_crd_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/crd.yaml"
  cluster_autoscaler_manifest_url: "https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/hetzner/examples/cluster-autoscaler-run-on-master.yaml"

datastore:
  mode: etcd
  external_datastore_endpoint: postgres://....

schedule_workloads_on_masters: false

image: ubuntu-22.04

masters_pool:
  instance_type: cpx11
  instance_count: 3
  location: hel1

worker_node_pools:
- name: med-static
  instance_type: cpx31
  instance_count: 3
  location: hel1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 6

embedded_registry_mirror:
  enabled: true

post_create_commands:
 - echo 'network:\n  version:\ 2\n  ethernets:\n    enp7s0:\n      critical:\ true\n      nameservers:\n        addresses:\ [10.0.0.2]\n      routes:\n      - on-link:\ true\n        to:\ 0.0.0.0/0\n        via:\ 10.0.0.1' > /etc/netplan/50-cloud-init.yaml
 - sed -i 's/\\//g' /etc/netplan/50-cloud-init.yaml
 - sed -i 's/^nameserver.*/nameserver 10.0.0.2/' /etc/resolv.conf
 - netplan apply
 - apt update
 - apt upgrade -y
 - apt autoremove -y

[vitobotta]

Hi, this has been reported a couple of times before but I haven't had a chance to try and reproduce the problem yet. Can you share more details on how you have configured the network in Hetzner? The more details the better as they might help me understand where the problem might be.