# Troubleshooting

## Machine lifecycle

To get an immediate view of the state of MachineDeployments and Nodes, run:

```shell
kubectl -n kube-system get machinedeployment,machineset,machine,no -o wide
NAME                                      AGE   DELETED   REPLICAS   AVAILABLEREPLICAS   PROVIDER    OS       VERSION
machinedeployment.cluster.k8s.io/worker   59d             2          2                   openstack   ubuntu   1.30.1

NAME                                          AGE   DELETED   REPLICAS   AVAILABLEREPLICAS   MACHINEDEPLOYMENT   PROVIDER    OS       VERSION
machineset.cluster.k8s.io/worker-677cf94d4d   8d              0                              worker              openstack   ubuntu   1.30.1
machineset.cluster.k8s.io/worker-77c8c559d6   8d              2          2                   worker              openstack   ubuntu   1.30.1

NAME                                             AGE   DELETED   MACHINESET          ADDRESS        NODE                      PROVIDER    OS       VERSION
machine.cluster.k8s.io/worker-77c8c559d6-mknks   8d              worker-77c8c559d6   192.168.1.20   worker-77c8c559d6-mknks   openstack   ubuntu   1.30.1
machine.cluster.k8s.io/worker-77c8c559d6-rfd8r   8d              worker-77c8c559d6   192.168.1.9    worker-77c8c559d6-rfd8r   openstack   ubuntu   1.30.1

NAME                           STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node/worker-77c8c559d6-mknks   Ready    <none>   8d    v1.30.1   192.168.1.20                     Ubuntu 22.04.4 LTS   5.15.0-107-generic   containerd://1.6.33
node/worker-77c8c559d6-rfd8r   Ready    <none>   8d    v1.30.1   192.168.1.9                      Ubuntu 22.04.4 LTS   5.15.0-107-generic   containerd://1.6.33
```

Note:

- When the MachineDeployment has converged, all but one of its managed MachineSets have 0 replicas
- All Machines of a given MachineSet share the same name prefix
- Node names are the same as Machine names
- When the Machines have been fully provisioned, there's a Node object for each Machine

## Machine Events

To inspect a specific Machine, check its Events:

```shell
kubectl -n kube-system events --for machine/worker-77c8c559d6-wvlcg                                                                                                                                                            08/09 17:52
LAST SEEN               TYPE     REASON                     OBJECT                            MESSAGE
9m58s                   Normal   Created                    Machine/worker-77c8c559d6-wvlcg   Successfully created instance
7m26s (x5 over 9m53s)   Normal   InstanceFound              Machine/worker-77c8c559d6-wvlcg   Found instance at cloud provider, status: running
6m57s (x2 over 6m59s)   Normal   LabelsAnnotationsUpdated   Machine/worker-77c8c559d6-wvlcg   Successfully updated labels/annotations
```

## Get Node conditions

To see the status of different Node conditions, run:

```shell
kubectl describe node $node
```

## List Pods running on Node

To list all Pods scheduled on a particular Node, run:

```shell
kubectl get pods --all-namespaces --field-selector spec.nodeName=$node
```

## Delete Pods running on Node

In case the Node isn't responsive, you may choose to force the immediate deletion of all Pods on a Node:

```shell
kubectl delete pods --all-namespaces --field-selector spec.nodeName=$node --force
```

> [!CAUTION]
>
> With the `--force` flag, Kubernetes does not wait for Pods and their containers to stop. Applications may continue to run! Under normal circumstances, rely on Kubelet to tear down the Pods.

## Get remote access using SSH

> [!NOTE]
>
> **Prerequisites**
>
> - SSH key agent enabled in the cluster
> - SSH public key added to the cluster
> - SSH client uses the SSH key
> - Either of:
>
>   - Floating IPs enabled for the MachineDeployment
>   - Other host available as SSH jump host
> - Port 22 accessible (default, see [node networking](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/networking/#ingress>))

> [!NOTE]
>
> Adding an SSH key after MetaKube creates a Machine works when the Kubelet on the Node is running and healthy.

1. Get public IP of Node

   ```shell
   IP=$(kubectl get node $node -o jsonpath='{.status.addresses[?(@.type == "ExternalIP")].address}')
   echo $IP
   ```
2. Establish SSH session

   ```shell
   ssh ubuntu@$IP
   ```

   ```shell
   ssh core@$IP
   ```

## Remote access by escaping from a privileged Pod

> [!NOTE]
>
> **Prerequisites**
>
> - Kubelet running and healthy
> - Cluster network functioning (for forwarding stdin &amp; stdout)

> [!CAUTION]
>
> The Pod has full access to the Node! Make sure to verify the integrity of the tooling and container image that is used!

By creating a Pod with a privileged container that's sharing the PID namespace of the host, you can switch to the Kernel namespaces to the init process. To do this, you can use a tool like [node-shell](<https://github.com/kvaps/kubectl-node-shell/tree/master>).

## Inspect Kubelet logs

> [!NOTE]
>
> **Prerequisites**
>
> - Shell access on a Node
>
> See [SSH](<#get-remote-access-using-ssh>) or [node shell](<#remote-access-by-escaping-from-a-privileged-pod>).

To tail the logs of Kubelet, run:

```shell
journalctl -exu kubelet -f
```

## Inspect Node bootstrapping with OpenStack

It's possible to get some output of the initialization process of a Node. These logs may contain valuable information on how far the initialization has progressed or surface potential errors (for example, DNS).

To show the logs of a Node using the [OpenStack CLI](<https://www.syseleven.de/support/>):

```shell
openstack console log show $node
```

## Node Provisioning

The most important three phases during Node provisioning are:

1. Server creation

   To verify this:

   - Check Machine [Events](<#machine-events>)
   - Query the server with the [OpenStack CLI](<https://www.syseleven.de/support/>) directly

     ```shell
     openstack server show $node
     ```
2. Node initialization

   Issues during this phase become visible by inspecting [Kubelet logs](<#inspect-kubelet-logs>) or the [OpenStack console output](<#inspect-node-bootstrapping-with-openstack>).
3. Initialize Node daemons

   > **Note**
   >
   > The Node is already registered with the Kubernetes cluster at this point.

   Check:

   1. [Node conditions](<#get-node-conditions>)
   2. [Pods running on Node](<#list-pods-running-on-node>)
   3. Get their logs

      To get the logs of, for example, the Cilium Pod running on the particular Node, run:

      ```shell
      kubectl -n kube-system logs -l app=cilium --field-selector spec.nodeName="$node"
      ```

## Node deletion

The following steps help determine the reason the Node is not deleted:

1. Check if the Machine has a deletion timestamp (`DELETED` column):

   ```shell
   kubectl -n kube-system get machine worker-77c8c559d6-rfd8r
   NAME                      AGE   DELETED   MACHINESET          ADDRESS       NODE                      PROVIDER    OS       VERSION                                                                             
   worker-77c8c559d6-rfd8r   8d    91s       worker-77c8c559d6   192.168.1.9   worker-77c8c559d6-rfd8r   openstack   ubuntu   1.30.1
   ```
2. Check whether the Node drains

   MetaKube drains the Node by evicting its Pods (with exceptions such as DaemonSet Pods). The [eviction API](<https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/>) attempts to delete Pods via the API. For example, a Pod may block eviction when a matching PodDisruptionBudget does not allow any more disruptions.

   Try draining the Node:

   ```shell
   kubectl drain --ignore-daemonsets --delete-emptydir-data $node
   ```

   It may report that some Pods are not safe to evict, with the reason.

   To get a list of Pods running on a Node, see [above](<#list-pods-running-on-node>):

   Another reason the Node may not drain is when Pods do not leave the `Terminating` state. This may be because of an unresponsive Kubelet.
3. MetaKube cannot delete the server

   When the Node is fully drained but remains in the cluster, MetaKube may not delete the cloud server. In that case, check the Machine [Events](<#machine-events>) for errors.

## Autoscaling

For issues related to autoscaling see [documentation](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/autoscaling/#troubleshooting>).
