# Internal Cluster networking

## Pod to Pod

- All Pods without `spec.hostNetwork: true` are assigned an IP from the Pod network by the CNI plugin.
- All Pods in a Kubernetes cluster, if not restricted through Network Policies, can generally talk to each other on all ports.

### CNI

A CNI plugin has multiple responsibilities:

- IPAM: Manage the IP address space of the Pod network and assign IPs to Pods
- Enable layer 3 (IP) communication between Pods
- (Optional) Implement Kubernetes Network Policies

#### Supported CNI Plugins

Currently, "Canal" is the only CNI plugin the MetaKube Core supports. It is automatically installed in every cluster.

### CNI Plugin: Canal

Canal is a combination of Flannel and Calico. Both take over a subset of the above-mentioned responsibilities.

#### IPAM: "host-local"

Canal is configured to use "host-local" IPAM.

- The Pod network is divided into `/24` subnet slices. Each node has its own ("local") slice assigned, which can be found in a node's `spec.podCIDR`.
- Since this assignment is fixed for the lifetime of a node, one can statically determine which node the Pod lives on.

#### Layer 3 Connectivity: VXLAN

Flannel creates an overlay over the node network using VXLAN. Frames are encapsulated into UDP datagrams. They are then transported between nodes on UDP port `8472` and decapsulated on the receiving side.

- Flannel creates a `flannel.1` bridge connected to the main ethernet interface of the host. This bridge has an IP matching the `/24` Pod network slice and a `0` suffix, for example `172.25.3.0` when the slice for the node is `172.25.3.0/24`.

```shell
$ ip -d a show flannel.1
9: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 22:3d:4f:df:d7:d7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    vxlan id 1 local 192.168.1.44 dev ens3 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 172.25.3.0/32 brd 172.25.3.0 scope global flannel.1
        valid_lft forever preferred_lft forever
    inet6 fe80::203d:4fff:fedf:d7d7/64 scope link
        valid_lft forever preferred_lft forever
```

- Calico creates a pair of `veth` devices for each Pod. One in the Pod's network namespace (`eth0@ifx`) with the Pod's IP and one on the host side (`calixxxxxxxxxxx@ifx`) connected to the bridge.
- For each Pod network slice, there's a corresponding route to the bridge on the respective host using the `flannel.1` interface. This means, traffic to Pods is routed through the `flannel.1` interface (and thus encapsulated) and sent to the host where the Pod lives.

```shell
$ ip r | grep flannel
172.25.0.0/24 via 172.25.0.0 dev flannel.1 onlink
172.25.1.0/24 via 172.25.1.0 dev flannel.1 onlink
172.25.2.0/24 via 172.25.2.0 dev flannel.1 onlink
```

#### Network Policies: Calico

Calico implements Kubernetes Network Policies. You can limit ingress or egress of Pods to only certain destinations to harden your cluster.

For more information on Kubernetes Network Policies, see the [official Kubernetes documentation](<https://kubernetes.io/docs/concepts/services-networking/network-policies>).

## Services

Service networking is implemented by kube-proxy. There are two modes of kube-proxy: `iptables` (legacy) and `ipvs`.

> [!NOTE]
>
> **Info**
>
> `ipvs` is superior to `iptables` mode for performance and scaling. For this reason, `iptables` is no longer available for new clusters. To migrate your existing cluster to `ipvs`, please contact our [Support](<https://documentation.syseleven.de/en/support/index.md>).

For more information on Kubernetes Services, see the [official Kubernetes documentation](<https://kubernetes.io/docs/concepts/services-networking/service>).

## Pod to host or host to Pod

Pods can also directly communicate with other nodes, for example node ports or Pods running with `spec.hostNetwork: true` and vice-versa.

## References

- [Flannel](<https://github.com/flannel-io/flannel>)
- [Calico](<https://projectcalico.docs.tigera.io>)
