# Autoscaling

MetaKube allows you to scale the size of MachineDeployments automatically to adjust to your dynamic workloads.

## Enable autoscaling

You can enable autoscaling for a MachineDeployment using different clients.

Configure autoscaling by specifying the `min_replicas` and `max_replicas` fields:

```terraform
resource "metakube_node_deployment" "nodes" {
  spec {
    min_replicas = 0
    max_replicas = 5
  }
}
```

See [Create MachineDeployment](<https://documentation.syseleven.de/en/products/metakube-core/usage/machine-deployments/#create-machine-deployment>).

Add the following annotations to the MachineDeployment:

```yaml
metadata:
  annotations:
    cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
    cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "15"
```

## Behavior

### Scaling up

If autoscaling is [enabled](<#enable-autoscaling>), the autoscaler scales up the MachineDeployment when **all** the following conditions hold:

- The current number of replicas is below the max number of replicas

  1. Get current replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     ```
  2. Get configured max replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-max-size}'
     ```
- The scheduler cannot place Pods, leaving them in `Pending` state due to limited resources

  Get Pods in Pending state:

  ```shell
  kubectl -n $namespace get pod --field-selector spec.nodeName==""
  ```
- The Nodes managed by the MachineDeployment allow the Pods to schedule

  Inspect the MachineDeployment's [taints and labels](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/topology/#pod-placement>) and whether the Pod matches.
- Adding a new Node creates enough free capacity to accommodate the Pods

  Inspect the Pod's containers' requests:

  ```shell
  kubectl -n $namespace get pod $pod -o jsonpath='{..requests}'
  ```

  Their sum must stay below the node's [allocatable resources](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/resources/profiles/index.md>).

The autoscaler calculates the number of required Nodes to schedule all Pending Pods and updates the replica count of the MachineDeployment to match.

### Scaling down

If autoscaling is [enabled](<#enable-autoscaling>), the autoscaler scales down the MachineDeployment when **all** the following conditions hold:

- The current number of replicas exceeds the specified min number of replicas

  1. Get current replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     ```
  2. Get configured min replica count

     ```shell
     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-min-size}'
     ```
- The last scale-up happened longer than 2 minutes ago
- The Nodes stay below a 50% usage threshold

  To check usage of the Nodes:

  ```shell
  kubectl top no
  ```
- The scheduler can place all Pods on fewer nodes

  Pods may not allow eviction, for example, because of [PodDisruptionBudgets](<https://kubernetes.io/docs/tasks/run-application/configure-pdb/>).

The autoscaler simulates moving existing Pods to other Nodes and calculates candidates for removal. It removes underutilized Nodes one at a time.

### Scaling up from zero

MetaKube supports scaling MachineDeployments down to zero and up from zero. You may also use [taints](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/topology/#taints>) or rely on node [labels](<https://documentation.syseleven.de/en/products/metakube-core/concepts/nodes/topology/#labels-and-affinity>).

## Configuration

MetaKube runs the [cluster autoscaler](<https://github.com/kubernetes/autoscaler>) with the generic Cluster API provider plugin. The version always matches the cluster's minor Kubernetes version.

We run the autoscaler with the following configuration flags:

```text
--scan-interval=1m
--scale-down-delay-after-add=2m
--scale-down-unneeded-time=2m
--scale-down-unready-time=2m
--skip-nodes-with-local-storage=false
--enforce-node-group-min-size=true
```

> [!NOTE]
>
> **Info**
>
> We do not provide a way to change this configuration. If you encounter issues or have special requirements, please [contact](<mailto:support@syseleven.de>) us.

> [!WARNING]
>
> **Local Storage**
>
> Autoscaling does not suit workloads that use host local storage. Because of the flag `--skip-nodes-with-local-storage=false`, Nodes with Pods that use, for example, `hostPath` volumes may still be candidates for removal during scale-down. This decision aims to prevent false positives and unnecessary scale-down blocks that the alternative causes.

## Troubleshooting

If your MachineDeployment does not scale up or down, examine the conditions required for scaling [up](<#scaling-up>) or [down](<#scaling-down>) respectively.

> [!NOTE]
>
> **Info**
>
> When you cannot determine why the MachineDeployment does not scale up or down, please contact our [support](<mailto:support@syseleven.de>). Include the output of the above steps in your inquiry.

> [!NOTE]
>
> **Autoscaling and OpenStack Resources**
>
> The Cluster Autoscaler strictly manages the lifecycle of your Kubernetes resources. While we guarantee the consistency of Kubernetes objects during scale-up and scale-down events of a MachineDeployment, we cannot guarantee the complete removal of the associated OpenStack resources (such as compute instances, volumes, or ports). You are responsible for monitoring your OpenStack usage to prevent unintended billing for left-over resources.

## References

- [Cluster Autoscaler GitHub](<https://github.com/kubernetes-sigs/cluster-autoscaler>)
