Cgroups

The Linux Kernel feature "cgroups" distributes system resources such as CPU and memory.

A Pod requests resources through its spec.containers[*].resources fields.

MetaKube nodes use systemd as the cgroup manager. The manager organizes cgroups in a hierarchy of "slices".

Depending on the Pod's determined Quality of Service, the system places it directly under kubepods.slice or one level lower under kubepods-burstable.slice or kubepods-besteffort.slice:

$ systemctl status
...
├─init.scope
├─user.slice
│ └─user-<uid>.slice
├─system.slice
│ ├─containerd.service
│ ├─kubelet.service
│ └─<other system services>.service
└─kubepods.slice
  ├─kubepods-pod<id>.slice
  │ ├─cri-containerd-<container id>.scope
  │ │ └─<pid> <command>
  │ └─cri-containerd-<container id>.scope
  │   └─<pid> /pause
  ├─kubepods-burstable.slice
  │ └─kubepods-burstable-pod<pod id>.slice
  └─kubepods-besteffort.slice
    └─kubepods-besteffort-pod<pod id>.slice

CPU

The Linux Kernel's completely fair scheduler (CFS) handles CPU time allocation.

The scheduler considers a container's resources.requests.cpu during placement to ensure the node has enough CPUs available. That value also corresponds to the container's cgroup cpu.weight.

The weight determines a process thread's place in the CFS's weighted queue and thus how often the scheduler runs it on a CPU core.

Kubelet divides the available resources proportionately so that a container process receives at least the configured requested CPU time (from resources.requests.cpu). Any spare CPU (either not requested or when processes yield) is free for waiting processes to use. The scheduler distributes it again based on each process's cgroup CPU weight, up to their cpu.max value (from resources.limits.cpu).

Warning

Kubernetes doesn't prevent you from over-committing a node (limits > available resources). Containers with low CPU requests might get throttled and fail to answer requests including liveness or readiness probes.

Memory

The scheduler considers a container's resources.requests.memory during placement to ensure the node has enough space available. Beyond scheduling, the value has no direct effect on cgroup memory limits.

A container's resources.limits.memory corresponds to the cgroup's memory.max value. Once reached, the process gets "OOM" killed.

Warning

Kubernetes doesn't prevent you from over-committing a node (limits > available resources) and exhausting the total available RAM. If any higher level slice or the entire system runs out of memory, the system chooses and kills a process in that slice or the system respectively.