Cilium Cluster Mesh for Multi-Cluster Connectivity

Multi-cluster Kubernetes normally means giving up on the network model you carefully built: services are exposed through ingress, traffic takes a detour through a load balancer, and network policy stops at the cluster boundary. Cilium Cluster Mesh keeps the model intact across clusters — pods in one cluster can reach services in another directly, service discovery is transparent, and identity-based policy can be applied across the mesh. It is the same eBPF data path you already run, extended by connecting the clusters' datastores. This guide covers the prerequisites that trip people up, the setup sequence, and the operational realities of cross-cluster traffic.

What the mesh actually provides

Prerequisites, in order of how often they are missed

1. Pod CIDRs must not overlap. The mesh forwards pod IPs directly; two clusters using 10.244.0.0/16 will produce silent mis-delivery rather than an obvious error. Pick per-cluster CIDRs from the start — retrofitting is a re-IP exercise.
2. Each cluster needs a unique numeric cluster ID when sharing a datastore.
3. Apiserver access between clusters. The clusters' Cilium agents must be able to reach the remote apiserver (or the shared kvstore) for endpoint information.
4. The same Cilium identity model and no conflicting node/service CIDRs.
5. Routing between node networks — a VPN, VPC peering, or native routing with reachable node IPs. Cluster Mesh does not build the underlay for you; if nodes cannot reach each other's pod traffic path, nothing works.

Enabling the mesh

# with the Cilium CLI, on the first cluster
cilium clustermesh enable --service-type LoadBalancer
cilium clustermesh status --wait

# on each additional cluster: enable with the same service type, then connect
cilium clustermesh enable --service-type LoadBalancer
cilium clustermesh connect --destination-context cluster1
cilium clustermesh status

# verify the mesh state
cilium clustermesh status --wait
kubectl get ciliumnodes -o wide
kubectl get svc -A | grep -i shared

Tunnelling versus native routing matters for cross-cluster latency: with VXLAN tunnelling, cross-cluster traffic takes the tunnel path and may hairpin through a gateway, while native routing with a routable pod CIDR gives you direct paths. Most production meshes use native routing where the node networks are mutually reachable and tunnelling where they are not.

Service affinity: keeping traffic local

# annotation on the Service: prefer local backends, fail over to remote
# service.cilium.io/affinity: "local"
kubectl annotate svc frontend service.cilium.io/affinity=local
kubectl annotate svc frontend service.cilium.io/affinity-timeout=1m

Without affinity, a service with backends in two clusters will be load-balanced across both, which is exactly right for active/active and exactly wrong for a workload where cross-cluster latency dominates. Local affinity keeps traffic in-cluster while the local backends are healthy, then spills over — a pragmatic middle ground that gives you regional locality with disaster recovery behaviour built in.

Operating the mesh

cilium status --verbose | head -40
cilium bpf lb list | grep -c ""          # service backends programmed
cilium monitor --type drop                # is anything being dropped cross-cluster?
cilium policy get | head
hubble observe --from-cluster cluster1 --to-cluster cluster2 --last 50

Hubble is the tool that makes a mesh debuggable: with cluster awareness, a flow that leaves cluster one and fails in cluster two is visible as a single line with both endpoints and the verdict, which turns "the remote call times out" into "policy in cluster two dropped the flow from the frontend identity". Plan for two operational costs: encryption and identity across clusters need consistent configuration everywhere, and a cluster joining or leaving the mesh triggers a control-plane event you should alert on, because a mesh that is silently half-connected behaves like a partial outage that only shows up as latency.

Underlay design: the reachability problem you must solve yourself

Cluster Mesh connects control planes; it does not invent connectivity. Before enabling anything, decide how pod and node traffic will actually flow between clusters, because that choice determines latency, encryption options and how much of your existing routing you have to change.

The MTU consequence is worth stating plainly: a tunnel-based underlay reduces the usable pod MTU, and a mixed environment where one cluster uses tunnelling and another native routing is where MTU black holes appear. Set pod MTU explicitly and consistently across clusters, and test with large packets rather than small ones.

Security posture across a mesh

Connecting clusters extends your trust boundary: an identity from cluster A becomes usable for policy decisions in cluster B. Three controls make that defensible.

# policy that only permits the flow you intend, evaluated on identity
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-to-api
  namespace: shop
spec:
  endpointSelector:
    matchLabels: {app: api}
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.cilium.k8s.policy.cluster: cluster1
            k8s:app: frontend
      toPorts:
        - ports: [{port: "8080", protocol: TCP}]

First, enforce policy in both directions and in every cluster: a mesh where one cluster runs in permissive mode is a mesh with a hole. Second, encrypt cross-cluster traffic where the underlying network is not trusted — a tunnel underlay plus per-node encryption keys is the usual answer, and the CPU cost is worth measuring against the alternative of exposing pod traffic to a shared network. Third, treat the Cluster Mesh control plane as sensitive infrastructure: the agents exchange endpoint information, so their credentials and the apiserver paths they use need the same care as any cluster-admin-adjacent component.

Failure modes and how they present

A service resolves but has no remote backends. The mesh is up locally but the remote cluster's endpoints are not visible: check cilium clustermesh status and the remote cluster's readiness, then confirm the clusters can reach each other's apiservers. This is almost always a network or certificate problem rather than a Cilium configuration problem.

Intermittent cross-cluster timeouts. Suspect MTU, then asymmetric routing. Tunnelling on one side and native routing on the other is the configuration that produces this, and it is diagnosed by sending a full-size packet and watching for a silent drop rather than a rejection.

Policy applies locally but not remotely. Check the identity labels in the policy: cross-cluster policies must match on the cluster label as well as the application label, and a policy written only for the local cluster will not select remote endpoints.

A cluster is removed without being disconnected. The mesh keeps trying to reach it and status churns. Always disconnect cleanly (cilium clustermesh disconnect) before decommissioning a cluster, and alert on mesh membership changes so that a silent half-connection is visible.

cilium clustermesh status --wait
cilium status | grep -i clustermesh
hubble observe --from-cluster cluster1 --to-cluster cluster2 --last 20
kubectl get ciliumendpointslices -A | head

FAQ

Does Cluster Mesh replace a service mesh? No. Cilium handles connectivity, load balancing and network policy; a service mesh layer adds traffic shifting, retries and mTLS at the application protocol level. The two are complementary and are often deployed together.

Can clusters be in different clouds or regions? Yes, that is a normal deployment. Latency between regions becomes a design input — which is exactly what service affinity exists to manage.

What about overlapping pod CIDRs discovered too late? There is no clean fix. Renumbering a cluster's pod CIDR is a disruptive operation involving restarts; the alternative is to place the overlapping clusters behind gateways and accept that they cannot join the mesh directly.

Is adoption heavy? If the clusters already run Cilium with compatible versions, enabling the mesh is a small change — enable on each side, connect, verify. The work is in the prerequisites and in policy, not in the mesh itself.

Quick reference: prerequisites and setup

# prerequisites, verified before anything else
kubectl -n kube-system get pods -l k8s-app=cilium           # Cilium healthy in both
cilium status | grep -i "cluster id"                        # unique cluster IDs
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'    # no overlap

# enable and connect
cilium clustermesh enable --service-type LoadBalancer
cilium clustermesh status --wait
cilium clustermesh connect --destination-context cluster1
cilium clustermesh status

# verify discovery and policy across the boundary
kubectl get ciliumnodes -o wide
hubble observe --from-cluster cluster1 --to-cluster cluster2 --last 20

Write the CIDR check into a script and run it as part of cluster provisioning: overlapping pod CIDRs are the one mistake in this design that is effectively unfixable later without renumbering, and it costs ten lines of validation to prevent. The second check worth automating is cluster ID uniqueness when a shared datastore is used, since a duplicated ID produces symptoms that look like random traffic loss rather than like a configuration error.

What to alert on

Three signals cover the mesh's failure modes: a change in mesh membership (a cluster connecting or disconnecting unexpectedly), a rising number of cross-cluster drops, and a service whose remote backends disappear while local ones remain. The third is the subtlest and the most damaging, because the service keeps working locally and only reveals the problem when the local cluster is the one under stress — which is precisely when remote capacity was supposed to help.

Related Reading

Related reading: Cilium BGP Control Plane in Kubernetes Clusters, Kubernetes CNI Compared: Calico vs Cilium vs Flannel, Kubernetes Gateway API vs Ingress: Migration Guide

原文链接：https://docs.cilium.io/en/stable/network/clustermesh/intro/