Skip to content

Troubleshooting

Machine lifecycle

To get an immediate view of the state of MachineDeployments and Nodes, run:

kubectl -n kube-system get machinedeployment,machineset,machine,no -o wide
NAME                                      AGE   DELETED   REPLICAS   AVAILABLEREPLICAS   PROVIDER    OS       VERSION
machinedeployment.cluster.k8s.io/worker   59d             2          2                   openstack   ubuntu   1.30.1

NAME                                          AGE   DELETED   REPLICAS   AVAILABLEREPLICAS   MACHINEDEPLOYMENT   PROVIDER    OS       VERSION
machineset.cluster.k8s.io/worker-677cf94d4d   8d              0                              worker              openstack   ubuntu   1.30.1
machineset.cluster.k8s.io/worker-77c8c559d6   8d              2          2                   worker              openstack   ubuntu   1.30.1

NAME                                             AGE   DELETED   MACHINESET          ADDRESS        NODE                      PROVIDER    OS       VERSION
machine.cluster.k8s.io/worker-77c8c559d6-mknks   8d              worker-77c8c559d6   192.168.1.20   worker-77c8c559d6-mknks   openstack   ubuntu   1.30.1
machine.cluster.k8s.io/worker-77c8c559d6-rfd8r   8d              worker-77c8c559d6   192.168.1.9    worker-77c8c559d6-rfd8r   openstack   ubuntu   1.30.1

NAME                           STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node/worker-77c8c559d6-mknks   Ready    <none>   8d    v1.30.1   192.168.1.20                     Ubuntu 22.04.4 LTS   5.15.0-107-generic   containerd://1.6.33
node/worker-77c8c559d6-rfd8r   Ready    <none>   8d    v1.30.1   192.168.1.9                      Ubuntu 22.04.4 LTS   5.15.0-107-generic   containerd://1.6.33

Note:

  • When the MachineDeployment has converged, all but one of its managed MachineSets have 0 replicas
  • All Machines of a given MachineSet share the same name prefix
  • Node names are the same as Machine names
  • When the Machines have been fully provisioned, there's a Node object for each Machine

Machine Events

To inspect a specific Machine, check its Events:

kubectl -n kube-system events --for machine/worker-77c8c559d6-wvlcg                                                                                                                                                            08/09 17:52
LAST SEEN               TYPE     REASON                     OBJECT                            MESSAGE
9m58s                   Normal   Created                    Machine/worker-77c8c559d6-wvlcg   Successfully created instance
7m26s (x5 over 9m53s)   Normal   InstanceFound              Machine/worker-77c8c559d6-wvlcg   Found instance at cloud provider, status: running
6m57s (x2 over 6m59s)   Normal   LabelsAnnotationsUpdated   Machine/worker-77c8c559d6-wvlcg   Successfully updated labels/annotations

Get Node conditions

To see the status of different Node conditions, run:

kubectl describe node $node

List Pods running on Node

To list all Pods scheduled on a particular Node, run:

kubectl get pods --all-namespaces --field-selector spec.nodeName=$node

Delete Pods running on Node

In case the Node isn't responsive, you may choose to force the immediate deletion of all Pods on a Node:

kubectl delete pods --all-namespaces --field-selector spec.nodeName=$node --force

Caution

With the --force flag, Kubernetes does not wait for Pods and their containers to stop. Applications may continue to run! Under normal circumstances, rely on Kubelet to tear down the Pods.

Get remote access using SSH

Prerequisites

  • SSH key agent enabled in the cluster
  • SSH public key added to the cluster
  • SSH client uses the SSH key
  • Either of:
    • Floating IPs enabled for the MachineDeployment
    • Other host available as SSH jump host
  • Port 22 accessible (default, see node networking)

Note

Adding an SSH key after MetaKube creates a Machine works when the Kubelet on the Node is running and healthy.

  1. Get public IP of Node

    IP=$(kubectl get node $node -o jsonpath='{.status.addresses[?(@.type == "ExternalIP")].address}')
echo $IP

  2. Establish SSH session

    ssh ubuntu@$IP

    ssh core@$IP

Remote access by escaping from a privileged Pod

Prerequisites

  • Kubelet running and healthy
  • Cluster network functioning (for forwarding stdin & stdout)

Caution

The Pod has full access to the Node! Make sure to verify the integrity of the tooling and container image that is used!

By creating a Pod with a privileged container that's sharing the PID namespace of the host, you can switch to the Kernel namespaces to the init process. To do this, you can use a tool like node-shell.

Inspect Kubelet logs

Prerequisites

  • Shell access on a Node

See SSH or node shell.

To tail the logs of Kubelet, run:

journalctl -exu kubelet -f

Inspect Node bootstrapping with OpenStack

It's possible to get some output of the initialization process of a Node. These logs may contain valuable information on how far the initialization has progressed or surface potential errors (for example, DNS).

To show the logs of a Node using the OpenStack CLI:

openstack console log show $node

Node Provisioning

The most important three phases during Node provisioning are:

  1. Server creation

    To verify this:

    • Check Machine Events

    • Query the server with the OpenStack CLI directly

      openstack server show $node

  2. Node initialization

    Issues during this phase become visible by inspecting Kubelet logs or the OpenStack console output.

  3. Initialize Node daemons

    Note

    The Node is already registered with the Kubernetes cluster at this point.

    Check:

    1. Node conditions
    2. Pods running on Node

    3. Get their logs

      To get the logs of, for example, the Cilium Pod running on the particular Node, run:

      kubectl -n kube-system logs -l app=cilium --field-selector spec.nodeName="$node"

Node deletion

The following steps help determine the reason the Node is not deleted:

  1. Check if the Machine has a deletion timestamp (DELETED column):

    kubectl -n kube-system get machine worker-77c8c559d6-rfd8r
NAME                      AGE   DELETED   MACHINESET          ADDRESS       NODE                      PROVIDER    OS       VERSION                                                                             
worker-77c8c559d6-rfd8r   8d    91s       worker-77c8c559d6   192.168.1.9   worker-77c8c559d6-rfd8r   openstack   ubuntu   1.30.1

  2. Check whether the Node drains

    MetaKube drains the Node by evicting its Pods (with exceptions such as DaemonSet Pods). The eviction API attempts to delete Pods via the API. For example, a Pod may block eviction when a matching PodDisruptionBudget does not allow any more disruptions.

    Try draining the Node:

    kubectl drain --ignore-daemonsets --delete-emptydir-data $node

    It may report that some Pods are not safe to evict, with the reason.

    To get a list of Pods running on a Node, see above:

    Another reason the Node may not drain is when Pods do not leave the Terminating state. This may be because of an unresponsive Kubelet.

  3. MetaKube cannot delete the server

    When the Node is fully drained but remains in the cluster, MetaKube may not delete the cloud server. In that case, check the Machine Events for errors.

Autoscaling

For issues related to autoscaling see documentation.