Skip to content

Cluster internal networking

Pod to Pod: CNI

Pod to Pod communication is facilitated by a CNI plugin which has multiple responsibilities:

  • IPAM: Manage the IP address space of the Pod network and assign IPs to Pods
  • Enable layer 3 (IP) communication between Pods
  • Implement Network Policies
  • (Optional) Handle service networking

MetaKube supports Cilium as the CNI plugin.

IPAM

Cilium uses Kubernetes IPAM. This means:

  1. Kubernetes controller-manager assigns a distinct /24 slice of the Pod network to each Node in node.spec.podCIDRs.
  2. Cilium will assign an IP from this subnet to each Pod that doesn't use the host's network (pod.spec.hostNetwork: true)

Layer 3 Connectivity

As long as NetworkPolicies allow, any Pod can talk to any other in the Cluster directly, without encapsulation.

  • OpenStack cloud controller manager creates a route for each /24 slice via the respective node.
  • Cilium creates a pair of veth devices for each Pod. One in the Pod's network namespace with the Pod's IP (eth0@ifx) and one on the host side (lxcxxxxxxxxxxxx@ifx).
  • Cilium creates a route for each other Node's /24 slice via the default interface to eliminate the hop over the router.

Network Policy

Cilium implements Kubernetes NetworkPolicies as well as its own CiliumNetworkPolicy and ClusterWideCiliumNetworkPolicy.

For more information see the Cilium documentation.

Known issue with named ports

There's currently still an open issue when multiple NetworkPolicies use a named port with the same name but different port numbers. Cilium is not able to differentiate them and will choose one arbitrarily.

To circumvent this, prefer port numbers in NetworkPolicies.

Service networking

Cilium also takes over the role of kube-proxy and implements service networking. In most cases, it behaves just like with kube-proxy, with some small caveats.

DNAT-less

If a connection to a virtual ClusterIP originates from one of the Nodes, Cilium intercepts the bind sys call and replaces the destination address of the socket with that of an endpoint. This eliminates the need for DNAT.

Pod to host or host to Pod

Pods can also communicate directly with other Nodes, e.g. on Node ports or Pods running with spec.hostNetwork: true and vice versa.

References