Nodes don't join after upgrade to 1.19

Problem

MetaKube Nodes that are created after a minor upgrade to 1.19 don't join the cluster.

Cause

A race condition between the control plane components leads to some RBAC rules not getting created. The Kubelet on the machines fails to get a Node Lease.

Affected K8s versions

Currently, the only versions for which we've observed this issue, was on 1.19.* after upgrading from 1.18.*.

Mitigation

After switching to another 1.19.* patch version the machines should join almost immediately.

To make sure, the necessary RBAC rules get installed, we enforce, that before upgrading to 1.20.*, that you'd first upgrade to 1.19.7 and then 1.19.12.

Complete fix

We're working on a permanent fix for the root cause of this issue.