# Autoscaling

The load on your workloads may fluctuate over time. With horizontal node autoscaling, instead of a fixed number of nodes, the autoscaler will automatically scale the MachineDeployment to an appropriate number of replicas.

> [!NOTE]
>
> **Info**
>
> This is especially effective in combination with Pod autoscalers, such as the [HorizontalPodAutoscaler](<https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>) built into Kubernetes.

## Enable autoscaling

You can enable autoscaling for a MachineDeployment using different clients.

When using the [MetaKube Terraform provider](<https://registry.terraform.io/providers/syseleven/metakube/latest>), you configure autoscaling by specifying the `min_replicas` and `max_replicas` fields:

```terraform
resource "metakube_node_deployment" "nodes" {
spec {
    min_replicas = 0
    max_replicas = 5
}
}
```

Just check the mark "Use horizontal node autoscaling" when creating or editing a MachineDeployment to enable autoscaling.

Just add the following annotations to the MachineDeployment:

```yaml
metadata:
annotations:
    cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
    cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "15"
```

## Behavior

### Scaling up

If autoscaling is [enabled](<#enable-autoscaling>), the autoscaler will scale up the MachineDeployment if **all** the following conditions are met:

- The current number of replicas is lower than the maximum number of replicas

  1. Get current replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     ```
  2. Get configured max replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-max-size}'
     ```
- Pods can not be scheduled and remain in `Pending` state due to limited resources

  Get Pods in Pending state:

  ```shell
  kubectl -n $namespace get pod --field-selector spec.nodeName==""
  ```
- The Nodes managed by the MachineDeployment allow the Pods to be scheduled

  Inspect the MachineDeployment's [taints and labels](<https://documentation.syseleven.de/en/prior-docs/metakube-core/concepts/nodes/topology/#pod-placement>) and if the Pod matches.
- Adding a new Node will create enough free capacity to accommodate the Pods

  Inspect the Pod's containers' requests:

  ```shell
  kubectl -n $namespace get pod $pod -o jsonpath='{..requests}'
  ```

  Their sum must be smaller than the node's [allocatable resources](<https://documentation.syseleven.de/en/prior-docs/metakube-core/concepts/nodes/resources/#allocatable-resources>).

The autoscaler calculates the number of required Nodes to schedule all Pending Pods and will update the replica count of the MachineDeployment accordingly.

### Scaling down

If autoscaling is [enabled](<#enable-autoscaling>), the autoscaler will scale down the MachineDeployment if **all** the following conditions are met:

- The current number of replicas is higher than the specified minimum number of replicas

  1. Get current replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     ```
  2. Get configured min replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-min-size}'
     ```
- The last scale-up hasn't happened within the last 2 minutes
- The Nodes are utilized under a 50% threshold

  To check usage of the Nodes:

  ```shell
  kubectl top no
  ```
- All Pods can be scheduled on fewer nodes

  It could be that Pods are not allowed to be evicted for example because of [PodDisruptionBudgets](<https://kubernetes.io/docs/tasks/run-application/configure-pdb/>).

The autoscaler will simulate moving existing Pods to other Nodes and calculate candidates for removal. It will remove underutilized Nodes one at a time.

### Scaling up from zero

MetaKube supports scaling MachineDeployments down to zero and up from zero. You may also use [taints](<https://documentation.syseleven.de/en/prior-docs/metakube-core/concepts/nodes/topology/#taints>) or rely on node [labels](<https://documentation.syseleven.de/en/prior-docs/metakube-core/concepts/nodes/topology/#labels-affinity>).

## Configuration

MetaKube runs the [cluster autoscaler](<https://github.com/kubernetes/autoscaler>) with the generic Cluster API provider plugin. The version always matches the cluster's minor Kubernetes version.

We use the following additional configuration flags:

```text
--scan-interval=1m
--scale-down-delay-after-add=2m
--scale-down-unneeded-time=2m
--scale-down-unready-time=2m
--skip-nodes-with-local-storage=false
--enforce-node-group-min-size=true
```

> [!NOTE]
>
> **Info**
>
> We currently provide no way to change this configuration. If you encounter issues or have special requirements, please contact our [support](<https://documentation.syseleven.de/en/support/index.md>).
>
> Autoscaling is not suitable for workloads that use host local storage. Because of the flag `--skip-nodes-with-local-storage=false`, Nodes with Pods that use for example `hostPath` volumes, may still be considered candidates to remove when scaling down. This is a deliberate decision as in our experience, the alternative very often leads to false positives and unnecessarily blocks scale down.

## Troubleshooting

If your MachineDeployment isn't scaling up or down, carefully examine the conditions required for scaling [up](<#scaling-up>) or [down](<#scaling-down>) respectively.

> [!NOTE]
>
> **Info**
>
> If you still can't find a reason the MachineDeployment isn't scaled up or down, please contact our [support](<https://documentation.syseleven.de/en/support/index.md>). Please include the output of the above steps in your inquiry.

## References

- [Cluster Autoscaler GitHub](<https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler>)
