Autoscaling

MetaKube allows you to scale the size of MachineDeployments automatically to adjust to your dynamic workloads.

Enable autoscaling

You can enable autoscaling for a MachineDeployment using different clients.

TerraformUIkubectl

MetaKube Terraform Provider

Configure autoscaling by specifying the min_replicas and max_replicas fields:

resource "metakube_node_deployment" "nodes" {
  spec {
    min_replicas = 0
    max_replicas = 5
  }
}

See Create MachineDeployment.

Add the following annotations to the MachineDeployment:

metadata:
  annotations:
    cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
    cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "15"

Behavior

Scaling up

If autoscaling is enabled, the autoscaler scales up the MachineDeployment when all the following conditions hold:

• The current number of replicas is below the max number of replicas

  1. Get current replica count

     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     

  2. Get configured max replica count

     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-max-size}'
     

• The scheduler cannot place Pods, leaving them in Pending state due to limited resources

  Get Pods in Pending state:

  kubectl -n $namespace get pod --field-selector spec.nodeName==""
  

• The Nodes managed by the MachineDeployment allow the Pods to schedule

  Inspect the MachineDeployment's taints and labels and whether the Pod matches.

• Adding a new Node creates enough free capacity to accommodate the Pods

  Inspect the Pod's containers' requests:

  kubectl -n $namespace get pod $pod -o jsonpath='{..requests}'
  

  Their sum must stay below the node's allocatable resources.

The autoscaler calculates the number of required Nodes to schedule all Pending Pods and updates the replica count of the MachineDeployment to match.

Scaling down

If autoscaling is enabled, the autoscaler scales down the MachineDeployment when all the following conditions hold:

• The current number of replicas exceeds the specified min number of replicas

  1. Get current replica count

     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.spec.replicas}'
     

  2. Get configured min replica count

     kubectl -n kube-system get machinedeployment $name -o jsonpath='{.metadata.annotations.cluster\.k8s\.io/cluster-api-autoscaler-node-group-min-size}'
     

• The last scale-up happened longer than 2 minutes ago

• The Nodes stay below a 50% usage threshold

  To check usage of the Nodes:

  kubectl top no
  

• The scheduler can place all Pods on fewer nodes

  Pods may not allow eviction, for example, because of PodDisruptionBudgets.

The autoscaler simulates moving existing Pods to other Nodes and calculates candidates for removal. It removes underutilized Nodes one at a time.

Scaling up from zero

MetaKube supports scaling MachineDeployments down to zero and up from zero. You may also use taints or rely on node labels.

Configuration

MetaKube runs the cluster autoscaler with the generic Cluster API provider plugin. The version always matches the cluster's minor Kubernetes version.

We run the autoscaler with the following configuration flags:

--scan-interval=1m
--scale-down-delay-after-add=2m
--scale-down-unneeded-time=2m
--scale-down-unready-time=2m
--skip-nodes-with-local-storage=false
--enforce-node-group-min-size=true

Info

We do not provide a way to change this configuration. If you encounter issues or have special requirements, please contact us.

Local Storage

Autoscaling does not suit workloads that use host local storage. Because of the flag --skip-nodes-with-local-storage=false, Nodes with Pods that use, for example, hostPath volumes may still be candidates for removal during scale-down. This decision aims to prevent false positives and unnecessary scale-down blocks that the alternative causes.

Troubleshooting

If your MachineDeployment does not scale up or down, examine the conditions required for scaling up or down respectively.

Info

When you cannot determine why the MachineDeployment does not scale up or down, please contact our support. Include the output of the above steps in your inquiry.

Autoscaling and OpenStack Resources

The Cluster Autoscaler strictly manages the lifecycle of your Kubernetes resources. While we guarantee the consistency of Kubernetes objects during scale-up and scale-down events of a MachineDeployment, we cannot guarantee the complete removal of the associated OpenStack resources (such as compute instances, volumes, or ports). You are responsible for monitoring your OpenStack usage to prevent unintended billing for left-over resources.

References

• Cluster Autoscaler GitHub