# Node NotReady because of unresponsive Kubelet

## Problem

A Node status `NotReady` may leave Pods in `Terminating` state indefinitely.

## Background

See [Nodes](<https://kubernetes.io/docs/concepts/architecture/nodes/#condition>).

## Symptoms

The problem can usually be observed as follows:

1. Node enters `NotReady` state.
2. Node controller terminates all Pods running on that Node.
3. Pods don't get deleted and stay in the `Terminating` status.

## Causes

Among the possible causes are:

- High rate of Pod creation
- Many sidecar containers

## Mitigation

Pods that are stuck in `Terminating` won't be deleted automatically, if the Kubelet isn't working. To forcefully remove all Pods of a Node, run:

```shell
kubectl delete pods --all-namespaces --field-selector spec.nodeName="${NAME_NAME}"
```

This will remove the Pods from the Kubernetes storage (etcd). Containers belonging to these Pods will remain running on the broken Node. Since the underlying machine will be deleted, this most likely won't matter though.

## Prevention &amp; Best Practices

Make sure to adhere to the following best practices:

- Update and replace Nodes often (treat Nodes as cattle)
- Don't overprovision your Nodes
- Define resource limits on your Pods
