Skip to content

Node configuration

Node labels

Node objects in a MetaKube cluster come with a set of useful labels that can be used for debugging purposes or setting affinity/anti-affinity rules while deploying applications.

In addition the built-in node labels MetaKube adds the following additional labels:

machine-controller/owned-by

This label contains the ID of the machine object that manages the node. See Cluster API for details.

machine-controller/physical-host-id

This label contains the ID of the physical server where the node is running on. This is particularly useful in scenarios where you might want to spread pod replicas across different physical servers, using pod anti-affinity rules to increase availability of your application.

Previously used label machine-controller/host-id is deprecated and is going to be permanently deleted.

system/cluster

This label contains the MetaKube cluster ID.

system/project

This label contains the MetaKube project ID.

Container logging configuration

All container runtimes that are installed with our provided images limit their log file size to 100MB for each container created (for example Docker logging option max-size). The idea is to prevent your worker node's disks being filled up by logs created by the containers. To store and keep track of your logs for a longer period, we recommend installing additional software to accomplish this. For example Elasticsearch, Logstash and Kibana or Loki, Promtail and Grafana.

Node-problem-detector

MetaKube deploys a node-problem-detector DaemonSet into every cluster. Node-problem-detector monitors various node-related error states and adds Kubernetes events and node conditions to a node if a problem occurs. The monitored error states are things like kernel deadlocks, hung tasks and corrupted container images. Please note that events and node conditions by themselves don't influence the Kubernetes scheduler, which means they won't directly prevent Kubernetes from using the node. See the next section for one instance where a node condition is actually processed further and does prevent pods from being scheduled on a node.

Advanced Node Readiness Tracking

In stock Kubernetes setups, the kubelet (node daemon) does not track the availability of the pod networking (canal) or cluster DNS services on the node. Especially during node creation, this may lead to a situation where the kubelet marks the node "ready" too early, and thus pods are scheduled onto it that will then fail. This is particularly problematic for Job or CronJob pods.

To address this problem, we're adding a customization to the node-problem-detector DaemonSet that's deployed into every cluster. That customization will add a node condition with type==MetakubeNodeReady and status False to a node when the networking or DNS aren't functional. Additional logic in MetaKube tracks this condition and adds a taint node.syseleven.de/not-ready:NoSchedule to the node as long as the condition is not True. As a result, all nodes in MetaKube clusters will have a taint node.syseleven.de/not-ready:NoSchedule when they start up, which will be removed when we determine that the network and DNS on the node are really up and running.

Node, Service and Pod IP ranges

By default when creating a cluster, MetaKube automatically configures Kubernetes to use a sensible internal IP range for Pod and Service IPs as well as an internal IP range for the worker nodes in the cloud provider's software-defined-network.

The defaults are:

  • Node IPs: 192.168.1.0/24
  • Pod IPs: 172.25.0.0/16
  • Service IPs: 10.240.16.0/20

In some situations, for example when connecting Kubernetes clusters from different regions, you may want to change this.

Changing these defaults is possible in the extended options of the cluster creation wizard.