# Setup Cilium Clustermesh

This guide explains how to setup cilium clustermesh on top of MetaKube instances with cilium cni.

## Prerequisites

This tutorial requires:

- Access to two MetaKube clusters using IPv4.
- PodCIDR/NodeCIDR ranges in all clusters and all nodes must be non-conflicting and unique IP addresses.
- Nodes in all clusters must have IP connectivity between each other using the configured InternalIP for each node.
- Access to OpenStack with valid and sufficient credentials
- OpenStack Subnet

## Assumptions

- The kubeconfig files are called `kubeconfig-cluster-1` and `kubeconfig-cluster-2`

## Create 2 MetaKube clusters with non overlapping pod/node CIDRs

Cluster-1

```json
...
   "spec": {
        "cloud": {
            "openstack":{
                "subnetCIDR":"192.168.0.0/24",
...
            }
        }
        "cniPlugin": {
            "type": "cilium",
            "cilium": {
                "clustermesh": {
                    "enable": true,
                    "clusterID": 1,
                    "ipv4NativeRoutingCIDR": "172.16.0.0/16"
                }
            }
        },
        "clusterNetwork": {
          "proxyMode":"none",
          "pods": {
            "cidrBlocks":[
              "172.16.0.0/17"
            ]
          },
        }
...
    }
```

Cluster-2

```json
...
   "spec": {
        "cloud": {
            "openstack":{
                "subnetCIDR":"192.168.1.0/24",
...
            }
        }
        "cniPlugin": {
            "type": "cilium",
            "cilium": {
                "clustermesh": {
                    "enable": true,
                    "clusterID": 2,
                    "ipv4NativeRoutingCIDR": "172.16.0.0/16"
                }
            }
        },
        "clusterNetwork": {
          "proxyMode":"none",
          "pods": {
            "cidrBlocks":[
              "172.16.128.0/17"
            ]
          },
        }
...
    }
```

## Enable clustermesh on existing cluster

> [!NOTE]
>
> **Info**
>
> If you change the cluster ID and/or cluster name in a cluster with running workloads, you will need to restart all workloads. The cluster ID is used to generate the security identity and it will need to be re-created in order to establish access across clusters. [See](<https://docs.cilium.io/en/stable/network/clustermesh/clustermesh/>)

```shell
METAKUBE_CLUSTER_ID=clusterID-1
METAKUBE_PROJECT_ID=projectID
curl -X PATCH https://metakube.syseleven.de/api/v2/projects/${METAKUBE_PROJECT_ID}/clusters/${METAKUBE_CLUSTER_ID} -H "accept: application/json"  -H "Authorization: Bearer <TOKEN> -d "$(jq -n '{spec: {cniPlugin: {cilium: {clustermesh: {enable: true, clusterID: 1, ipv4NativeRoutingCIDR: "172.16.0.0/16"}}}}}')"
```

## Connect to the clusters

## Copy `cilium-ca`

1. copy `cilium-ca` over to the second cluster

   ```shell
   kubectl get secret -n kube-system cilium-ca -o yaml | \
   kubectl neat | \
   kubectl --kubeconfig kubeconfig-cluster-2 apply -f -
   ```

> [!NOTE]
>
> **Info**
>
> This is also recommended by cilium if you are planning to run Hubble Relay across clusters, it is best to share a certificate authority (CA) between the clusters as it will enable mTLS across clusters to just work. [See](<https://docs.cilium.io/en/stable/network/clustermesh/clustermesh/#shared-certificate-authority>)

2. restart `clustermesh-apiserver` in peer cluster

   ```shell
   kubectl --kubeconfig kubeconfig-cluster-2 -n kube-system rollout restart deployment clustermesh-apiserver
   ```

## Connect the cluster networks

> [!NOTE]
>
> **Info**
>
> Needs to be torn down before cluster deletion. See section below.

### Clusters in the same region ( for different regions see below )

```shell
openstack network create interconn
openstack subnet create --network interconn --subnet-range 10.0.0.0/30 --gateway none interconn
```

cluster-1

```shell
METAKUBE_CLUSTER_ID=clusterID-1
openstack port create --network interconn --fixed-ip subnet=interconn,ip-address=10.0.0.1 ic-${METAKUBE_CLUSTER_ID}
openstack router add port metakube-${METAKUBE_CLUSTER_ID} ic-${METAKUBE_CLUSTER_ID}
openstack router add route --route destination=192.168.1.0/24,gateway=10.0.0.2 metakube-${METAKUBE_CLUSTER_ID}
openstack router add route --route destination=172.16.128.0/17,gateway=10.0.0.2 metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '192.168.1.0/24' metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '172.16.0.0/16' metakube-${METAKUBE_CLUSTER_ID}
```

cluster-2

```shell
METAKUBE_CLUSTER_ID=clusterID-2
openstack port create --network interconn --fixed-ip subnet=interconn,ip-address=10.0.0.2 ic-${METAKUBE_CLUSTER_ID}
openstack router add port metakube-${METAKUBE_CLUSTER_ID} ic-${METAKUBE_CLUSTER_ID}
openstack router add route --route destination=192.168.0.0/24,gateway=10.0.0.1 metakube-${METAKUBE_CLUSTER_ID}
openstack router add route --route destination=172.16.0.0/17,gateway=10.0.0.1 metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '192.168.0.0/24' metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '172.16.0.0/16' metakube-${METAKUBE_CLUSTER_ID}
```

### Clusters in different regions

cluster-1

```shell
METAKUBE_CLUSTER_ID=clusterID-1
openstack security group rule create --ingress --remote-ip '192.168.1.0/24' metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '172.16.0.0/16' metakube-${METAKUBE_CLUSTER_ID}
openstack subnet create --network metakube-${METAKUBE_CLUSTER_ID} --subnet-range 172.16.0.0/17 --gateway none metakube-${METAKUBE_CLUSTER_ID}-pod
openstack port create --network metakube-${METAKUBE_CLUSTER_ID} --fixed-ip subnet=metakube-${METAKUBE_CLUSTER_ID}-pod,ip-address=172.16.127.254 metakube-${METAKUBE_CLUSTER_ID}-pod
openstack router add port metakube-${METAKUBE_CLUSTER_ID} metakube-${METAKUBE_CLUSTER_ID}-pod

openstack vpn ike policy create metakube-${METAKUBE_CLUSTER_ID} --ike-version v2 --auth-algorithm sha256 --encryption-algorithm aes-256 --pfs group14
openstack vpn ipsec policy create metakube-${METAKUBE_CLUSTER_ID} --auth-algorithm sha256 --encryption-algorithm aes-256 --pfs group14
openstack vpn service create metakube-${METAKUBE_CLUSTER_ID} --router metakube-${METAKUBE_CLUSTER_ID}

openstack vpn endpoint group create metakube-${METAKUBE_CLUSTER_ID}-local-epg --type subnet --value metakube-${METAKUBE_CLUSTER_ID} --value metakube-${METAKUBE_CLUSTER_ID}-pod
openstack vpn endpoint group create metakube-${METAKUBE_CLUSTER_ID}-peer-epg --type cidr --value 192.168.1.0/24 --value 172.16.128.0/17
```

cluster-2

```shell
METAKUBE_CLUSTER_ID=clusterID-2
openstack security group rule create --ingress --remote-ip '192.168.0.0/24' metakube-${METAKUBE_CLUSTER_ID}
openstack security group rule create --ingress --remote-ip '172.16.0.0/16' metakube-${METAKUBE_CLUSTER_ID}
openstack subnet create --network metakube-${METAKUBE_CLUSTER_ID} --subnet-range 172.16.128.0/17 --gateway none metakube-${METAKUBE_CLUSTER_ID}-pod
openstack port create --network metakube-${METAKUBE_CLUSTER_ID} --fixed-ip subnet=metakube-${METAKUBE_CLUSTER_ID}-pod,ip-address=172.16.255.254 metakube-${METAKUBE_CLUSTER_ID}-pod
openstack router add port metakube-${METAKUBE_CLUSTER_ID} metakube-${METAKUBE_CLUSTER_ID}-pod

openstack vpn ike policy create metakube-${METAKUBE_CLUSTER_ID} --ike-version v2 --auth-algorithm sha256 --encryption-algorithm aes-256 --pfs group14
openstack vpn ipsec policy create metakube-${METAKUBE_CLUSTER_ID} --auth-algorithm sha256 --encryption-algorithm aes-256 --pfs group14
openstack vpn service create metakube-${METAKUBE_CLUSTER_ID} --router metakube-${METAKUBE_CLUSTER_ID}

openstack vpn endpoint group create metakube-${METAKUBE_CLUSTER_ID}-local-epg --type subnet --value metakube-${METAKUBE_CLUSTER_ID} --value metakube-${METAKUBE_CLUSTER_ID}-pod
openstack vpn endpoint group create metakube-${METAKUBE_CLUSTER_ID}-peer-epg --type cidr --value 192.168.0.0/24 --value 172.16.0.0/17
```

cluster-1

Grab the public IP of the OpenStack VPN service from `cluster-2`:

```shell
METAKUBE_CLUSTER_ID=clusterID-2
PEER_ADDRESS=$(openstack vpn service show metakube-${METAKUBE_CLUSTER_ID} -f value -c external_v4_ip)
```

and connect the VPN site from `cluster-1` side:

```shell
PEER_METAKUBE_CLUSTER_ID=clusterID-2
openstack vpn ipsec site connection create metakube-${PEER_METAKUBE_CLUSTER_ID} \
  --vpnservice metakube-${METAKUBE_CLUSTER_ID} \
  --ikepolicy metakube-${METAKUBE_CLUSTER_ID} \
  --ipsecpolicy metakube-${METAKUBE_CLUSTER_ID} \
  --local-endpoint-group metakube-${METAKUBE_CLUSTER_ID}-local-epg \
  --local-id left-peer.domain.example \
  --peer-address ${PEER_ADDRESS} \
  --peer-id right-peer.domain.example \
  --peer-endpoint-group metakube-${METAKUBE_CLUSTER_ID}-peer-epg \
  --psk secret
```

cluster-2

Grab the public IP of the OpenStack VPN service from `cluster-1`:

```shell
METAKUBE_CLUSTER_ID=clusterID-1
PEER_ADDRESS=$(openstack vpn service show metakube-${METAKUBE_CLUSTER_ID} -f value -c external_v4_ip)
```

and connect the VPN site from `cluster-2` side:

```shell
PEER_METAKUBE_CLUSTER_ID=clusterID-1
openstack vpn ipsec site connection create metakube-${PEER_METAKUBE_CLUSTER_ID} \
  --vpnservice metakube-${METAKUBE_CLUSTER_ID} \
  --ikepolicy metakube-${METAKUBE_CLUSTER_ID} \
  --ipsecpolicy metakube-${METAKUBE_CLUSTER_ID} \
  --local-endpoint-group metakube-${METAKUBE_CLUSTER_ID}-local-epg \
  --local-id right-peer.domain.example \
  --peer-address ${PEER_ADDRESS} \
  --peer-id left-peer.domain.example \
  --peer-endpoint-group metakube-${METAKUBE_CLUSTER_ID}-peer-epg \
  --psk secret
```

## Create externalName services

cluster-1

```shell
LB_IP=$(kubectl --kubeconfig kubeconfig-cluster-2 get -n kube-system service clustermesh-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
kubectl -n kube-system create service externalname cluster-${PEER_METAKUBE_CLUSTER_ID} --external-name ${LB_IP}.nip.io
```

cluster-2

```shell
LB_IP=$(kubectl --kubeconfig kubeconfig-cluster-1 get -n kube-system service clustermesh-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
kubectl -n kube-system create service externalname cluster-${PEER_METAKUBE_CLUSTER_ID} --external-name ${LB_IP}.nip.io
```

## Connect the clusters

cluster-1

```shell
PEER_METAKUBE_CLUSTER_ID=clusterID-2
helm template cilium cilium/cilium \
  --version v1.18.7 \
  --namespace kube-system \
  --set cluster.name=${METAKUBE_CLUSTER_ID} \
  --set cluster.id=1 \
  --set clustermesh.useAPIServer=true \
  --set clustermesh.config.enabled=true \
  --set clustermesh.apiserver.tls.authMode=cluster \
  --set clustermesh.apiserver.kvstoremesh.enabled=true \
  --set clustermesh.apiserver.service.type=LoadBalancer \
  --set 'clustermesh.config.clusters[0].name='${PEER_METAKUBE_CLUSTER_ID} \
  --set 'clustermesh.config.clusters[0].port=2379' \
  --set 'clustermesh.config.clusters[0].address=cluster-'${PEER_METAKUBE_CLUSTER_ID}'.kube-system' |\
yq -cr -y 'select((.kind == "Secret") and (.metadata.name | test("cilium-(clustermesh|kvstoremesh)")))' | \
kubectl apply -f -
```

cluster-2

```shell
PEER_METAKUBE_CLUSTER_ID=clusterID-1
helm template cilium cilium/cilium \
  --version v1.18.7 \
  --namespace kube-system \
  --set cluster.name=${METAKUBE_CLUSTER_ID} \
  --set cluster.id=2 \
  --set clustermesh.useAPIServer=true \
  --set clustermesh.config.enabled=true \
  --set clustermesh.apiserver.tls.authMode=cluster \
  --set clustermesh.apiserver.kvstoremesh.enabled=true \
  --set clustermesh.apiserver.service.type=LoadBalancer \
  --set 'clustermesh.config.clusters[0].name='${PEER_METAKUBE_CLUSTER_ID} \
  --set 'clustermesh.config.clusters[0].port=2379' \
  --set 'clustermesh.config.clusters[0].address=cluster-'${PEER_METAKUBE_CLUSTER_ID}'.kube-system' |\
yq -cr -y 'select((.kind == "Secret") and (.metadata.name | test("cilium-(clustermesh|kvstoremesh)")))' | \
kubectl apply -f -
```

## Tear cluster network connection down

cluster-1

```shell
METAKUBE_CLUSTER_ID=clusterID-1
openstack vpn ipsec site connection delete $(openstack vpn ipsec site connection list -f value -c Name)
openstack vpn endpoint group delete metakube-${METAKUBE_CLUSTER_ID}-local-epg
openstack vpn endpoint group delete metakube-${METAKUBE_CLUSTER_ID}-peer-epg
openstack vpn service delete metakube-${METAKUBE_CLUSTER_ID}
openstack vpn ike policy delete metakube-${METAKUBE_CLUSTER_ID}
openstack vpn ipsec policy delete metakube-${METAKUBE_CLUSTER_ID}
openstack router remove port metakube-${METAKUBE_CLUSTER_ID} metakube-${METAKUBE_CLUSTER_ID}-pod
openstack subnet delete metakube-${METAKUBE_CLUSTER_ID}-pod
```

cluster-2

```shell
METAKUBE_CLUSTER_ID=clusterID-2
openstack vpn ipsec site connection delete $(openstack vpn ipsec site connection list -f value -c Name)
openstack vpn endpoint group delete metakube-${METAKUBE_CLUSTER_ID}-local-epg
openstack vpn endpoint group delete metakube-${METAKUBE_CLUSTER_ID}-peer-epg
openstack vpn service delete metakube-${METAKUBE_CLUSTER_ID}
openstack vpn ike policy delete metakube-${METAKUBE_CLUSTER_ID}
openstack vpn ipsec policy delete metakube-${METAKUBE_CLUSTER_ID}
openstack router remove port metakube-${METAKUBE_CLUSTER_ID} metakube-${METAKUBE_CLUSTER_ID}-pod
openstack subnet delete metakube-${METAKUBE_CLUSTER_ID}-pod
```

## Troubleshooting

1. cilium endpoint

   ```shell
   kubectl -n kube-system exec -it daemonset/cilium -- cilium-dbg endpoint list | grep app=http-echo
   622        Disabled           Disabled          66711      k8s:app=http-echo
   ```

   If you don't see the expected endpoint try to roll overt the cilium DS

   ```shell
   kubectl -n kube-system rollout restart daemonset cilium
   ```
2. clustermesh service

   ```shell
   kubectl -n kube-system exec -ti deployment/clustermesh-apiserver -c apiserver -- clustermesh-apiserver shell kvstore/list | grep -E "# cilium/(cache|state)/services/"
   # cilium/cache/services/v1/clusterID-2/default/http-echo
   # cilium/state/services/v1/clusterID-1/default/http-echo
   ```
3. cilium agent connecting to the local kvstoremesh

   ```shell
   kubectl -n kube-system exec -it daemonset/cilium -- cilium-dbg troubleshoot clustermesh
       Found 1 cluster configurations

       Cluster "clusterID-2":
       📄 Configuration path: /var/lib/cilium/clustermesh/clusterID-2

       🔌 Endpoints:
       - https://clustermesh-apiserver.kube-system.svc:2379
           ✅ Hostname resolved to: 10.240.22.125
           ✅ TCP connection successfully established to 10.240.22.125:2379
           ✅ TLS connection successfully established to 10.240.22.125:2379
           ℹ️  Negotiated TLS version: TLS 1.3, ciphersuite TLS_AES_128_GCM_SHA256
           ℹ️  Etcd server version: 3.6.6

       🔑 Digital certificates:
       ✅ TLS Root CA certificates:
           - Serial number:       fb:62:5c:15:b9:20:12:e7:70:3b:6d:12:ee:7d:cc:14
               Subject:             CN=Cilium CA
               Issuer:              CN=Cilium CA
               Validity:
               Not before:  2026-01-12 15:25:04 +0000 UTC
               Not after:   2029-01-11 15:25:04 +0000 UTC
       ✅ TLS client certificates:
           - Serial number:       2c:41:b7:b3:0d:94:29:67:6a:c1:80:9b:2a:ea:e7:14
               Subject:             CN=local-clusterID-1
               Issuer:              CN=Cilium CA
               Validity:
               Not before:  2026-01-12 16:03:35 +0000 UTC
               Not after:   2029-01-11 16:03:35 +0000 UTC

       ⚙️ Etcd client:
       ✅ Etcd connection successfully established
       ℹ️  Etcd cluster ID: f159592cd2603408
   ```
4. kvstoremesh connecting to peer cluster

   ```shell
   kubectl -n kube-system exec -ti deployment/clustermesh-apiserver -c kvstoremesh -- clustermesh-apiserver kvstoremesh-dbg troubleshoot
       Found 1 cluster configurations

       Cluster "clusterID-2":
       📄 Configuration path: /var/lib/cilium/clustermesh/clusterID-2

       🔌 Endpoints:
       - https://clusterID-2.mesh.cilium.io:2379
           ✅ Hostname resolved to: 109.68.229.69
           ✅ TCP connection successfully established to 109.68.229.69:2379
           ✅ TLS connection successfully established to 109.68.229.69:2379
           ℹ️  Negotiated TLS version: TLS 1.3, ciphersuite TLS_AES_128_GCM_SHA256
           ℹ️  Etcd server version: 3.6.6

       🔑 Digital certificates:
       ✅ TLS Root CA certificates:
           - Serial number:       fb:62:5c:15:b9:20:12:e7:70:3b:6d:12:ee:7d:cc:14
               Subject:             CN=Cilium CA
               Issuer:              CN=Cilium CA
               Validity:
               Not before:  2026-01-12 15:25:04 +0000 UTC
               Not after:   2029-01-11 15:25:04 +0000 UTC
       ✅ TLS client certificates:
           - Serial number:       d2:77:54:8b:e6:b4:91:3d:cd:b5:3f:fe:ad:b3:d2:84
               Subject:             CN=remote
               Issuer:              CN=Cilium CA
               Validity:
               Not before:  2026-01-12 16:03:35 +0000 UTC
               Not after:   2029-01-11 16:03:35 +0000 UTC

       ⚙️ Etcd client:
       ✅ Etcd connection successfully established
       ℹ️  Etcd cluster ID: 7711876c9ebb329e
   ```
5. overall cluster health

   ```shell
   kubectl -n kube-system exec -it daemonset/cilium -- cilium-dbg status --all-clusters
       KVStore:                Disabled
       Kubernetes:             Ok         1.34 (v1.34.2) [linux/amd64]
       Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumCIDRGroup", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Pods", "networking.k8s.io/v1::NetworkPolicy"]
       KubeProxyReplacement:   True   [ens3   192.168.0.10 fe80::f816:3eff:fe26:8da8 (Direct Routing)]
       Host firewall:          Disabled
       SRv6:                   Disabled
       CNI Chaining:           none
       CNI Config file:        successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
       Cilium:                 Ok   1.18.4 (v1.18.4-afda2aa9)
       NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
       Cilium health daemon:   Ok
       IPAM:                   IPv4: 9/254 allocated from 172.16.1.0/24,
       ClusterMesh:            1/1 remote clusters ready, 1 global-services
       clusterID-2: ready, 2 nodes, 19 endpoints, 0 identities, 1 services, 0 MCS-API service exports, 0 reconnections (last: never)
       └  etcd: 1/1 connected, leases=0, lock leases=0, has-quorum=true: endpoint status checks are disabled, ID: f159592cd2603408
       └  remote configuration: expected=true, retrieved=true, cluster-id=2, kvstoremesh=true, sync-canaries=true, service-exports=disabled
       └  synchronization status: nodes=true, endpoints=true, identities=true, services=true
       IPv4 BIG TCP:            Disabled
       IPv6 BIG TCP:            Disabled
       BandwidthManager:        Disabled
       Routing:                 Network: Native   Host: BPF
       Attach Mode:             TCX
       Device Mode:             veth
       Masquerading:            BPF   [ens3]   172.16.0.0/16  [IPv4: Enabled, IPv6: Disabled]
       Controller Status:       59/59 healthy
       Proxy Status:            OK, ip 172.16.1.149, 0 redirects active on ports 10000-20000, Envoy: external
       Global Identity Range:   min 65536, max 131071
       Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 36.02   Metrics: Ok
       Encryption:              Disabled
       Cluster health:          4/4 reachable   (2026-01-13T09:02:48Z)   (Probe interval: 2m11.83347464s)
       Name                     IP              Node                     Endpoints
       Modules Health:          Stopped(16) Degraded(3) OK(84)
   ```
6. try etcdctl

```shell
    export PEER_CLUSTER=clusterID-2 ; etcdctl \
  --endpoints=$(kubectl -n kube-system get secret cilium-kvstoremesh -o json | jq -cr '.data | with_entries(select(.key | test("etcd") | not)) | ."'${PEER_CLUSTER}'" |  @base64d' | yq -c ".endpoints?[]") \
  --cacert=<(kubectl -n kube-system get secret clustermesh-apiserver-remote-cert -o json | jq -cr '.data."ca.crt" |  @base64d') \
  --cert =(kubectl -n kube-system get secret clustermesh-apiserver-remote-cert -o json | jq -cr '.data."tls.crt" |  @base64d') \
  --key =(kubectl -n kube-system get secret clustermesh-apiserver-remote-cert -o json | jq -cr '.data."tls.key" |  @base64d') \
 get --prefix cilium/state/
```
