# Known Issues

## Idle TCP sessions being closed

> [!NOTE]
>
> **Info**
>
> This guide is an extension of the [known Issues documentation](<https://documentation.syseleven.de/en/prior-docs/openstack-cloud/reference/network/known-issues/index.md>) from OpenStack. The issue in MetaKube is related and quite similar, but needs to be handled slightly different.

### Affected Regions

CBK and DBL

### Problem Statement

MetaKube worker nodes run as virtual machines on OpenStack Cloud.

When a virtual machine establishes a TCP connection to a remote server, it uses a random TCP source port. In order for return traffic to be allowed to flow into a VM in OpenStack, a dynamic inbound security group rule will automatically be created by the SDN (Software Defined Network), allowing traffic to flow back to this random TCP port. This dynamic rule will expire if the connection is idle for 60 seconds.

If the server is quiet for too long, any return traffic from the remote server will be dropped.

### Solution

Follow these steps to avoid this issue:

### OpenStack

Select one of the following OpenStack solutions which applies for your setup:

- Either add a rule to your existing MetaKube security group, which explicitly allows returning traffic. Now the SDN has no need to create dynamic rules anymore. The Linux kernel option `net.ipv4.ip_local_port_range` configures the range from which the random source port will be picked when a virtual machine initiates a connection. For example, setting this value to `32768 - 60999` and allowing all traffic incoming from the server to the client port range `32768 - 60999` will solve the issue. To determine `net.ipv4.ip_local_port_range` for your worker nodes you need SSH access to the VM or connect through a [node-shell](<https://github.com/kvaps/kubectl-node-shell>) and issue: `sysctl net.ipv4.ip_local_port_range`
- or just in case both virtual machines (the client application and the server) run in the same OpenStack project and region, add a security group to the server which allows ingress traffic to high ports such as `32768 - 60999` from another security group that the server's ports are members of.

## MetaKube

- Your application **must** support enabling TCP keepalives. Turn it on with a timeout value shorter than `60` seconds.
- The following (or similar) sysctl parameters **must** be set in the network namespace of the **container/Pod** (not of the host):
- `net.ipv4.tcp_keepalive_intvl = 10`
- `net.ipv4.tcp_keepalive_probes = 5`
- `net.ipv4.tcp_keepalive_time = 10`

Kubelet does not consider these settings as "safe" and will not allow containers to run, if they are set in `podSecurityContext` [section](<https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod>) of your container. Currently, MetaKube does not offer a way to change this.

A valid alternative is to set them in an `initContainer` in the same Pod as your application.

Example:

```YAML
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: myapp-client
  name: myapp-client
spec:
  containers:
  - name: myapp-client
    image: busybox
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:  
  - command: [sh, -c]
    args:  
      - |  
        echo "10" > /proc/sys/net/ipv4/tcp_keepalive_intvl
        echo "5" > /proc/sys/net/ipv4/tcp_keepalive_probes
        echo "10" > /proc/sys/net/ipv4/tcp_keepalive_time
    image: alpine:3.16  
    name: sysctl  
    securityContext:  
      runAsUser: 0  
      runAsGroup: 0  
      privileged: true
```

Verify the settings by: `kubectl exec -it myapp-client -- sysctl -a | grep keepalive`

Since keepalives are now activated but not automatically sent on every TCP connection, still the application must request kernel keepalives when it opens the TCP socket. To take effect, the MetaKube solution steps must be used in combination.

## References

- [OpenStack Known Issues documentation](<https://documentation.syseleven.de/en/prior-docs/openstack-cloud/reference/network/known-issues/index.md>)
- [kubectl node-shell plugin](<https://github.com/kvaps/kubectl-node-shell>)
- [podSecurityContext](<https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod>)
- [initContainers](<https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#init-containers-in-use>)
