Kubernetes CNI Compared: Calico vs Cilium vs Flannel

The CNI plugin is the most expensive networking decision in a Kubernetes cluster, and it is usually made once and never revisited. All three mainstream options implement the same requirement - every pod gets a routable IP with no NAT between pods - but they program the kernel very differently. This comparison covers the data plane each one builds, what that costs, and the policy features that decide multi-tenant use cases.

The Three Rules Every CNI Must Satisfy

1. Every pod gets a unique IP address that is routable from every other pod without NAT.
2. Containers in the same pod share a network namespace, so they reach each other over localhost.
3. The kubelet invokes the CNI plugin at pod setup to wire the interface; the plugin then gets out of the way, and packets are forwarded by kernel routes, iptables or eBPF programs.

Point three is the key insight for performance debates: CNI throughput is kernel data-plane throughput. The plugin binary's own speed is irrelevant once the pod is up.

Comparison at a Glance

The numbers in that table are the whole argument in miniature: the two options that route pods instead of encapsulating them recover the overlay tax, and Cilium buys that recovery with eBPF and a different debugging model.

Flannel - Overlay, Nothing More

# k3s: leave the CNI slot open for Calico or Cilium
--flannel-backend=none --disable-network-policy

Flannel allocates pod subnets per node and encapsulates traffic with VXLAN (or host-gw on a flat L2). It is portable, simple, and has no built-in NetworkPolicy support - so multi-tenancy needs a second plugin for policy. VXLAN encapsulation adds roughly 50 bytes of header per packet and, in published comparisons, caps throughput around 6-7 Gbps on 10G interfaces against roughly 9-10 Gbps for routed alternatives.

That 50-byte figure is worth unpacking, because it explains both the throughput loss and an MTU failure mode that catches people out. A VXLAN-tunnelled packet carries an outer Ethernet header (14), an outer IP header (20), a UDP header (8) and the VXLAN header (8) - 50 bytes on top of the inner frame. If the physical interface MTU is 1500 and the pod netns is left at 1500 as well, every full-size packet must be either fragmented or dropped. The correct configuration sets the pod interface MTU to the physical MTU minus the encapsulation overhead:

outer overhead (VXLAN)  = 14 + 20 + 8 + 8 = 50 bytes
pod MTU on a 1500 link = 1500 - 50 = 1450
pod MTU on a 9000 link = 9000 - 50 = 8950

Flannel is genuinely the right answer for labs, single-tenant clusters and k3s edge deployments where connectivity - not policy - is the requirement.

Calico - Route the Pods

Calico treats the cluster as a routable network. On bare metal and BGP-capable environments it advertises pod CIDRs to the physical fabric rather than encapsulating, which removes the overlay tax and keeps pod traffic inspectable with ordinary routing commands.

Node-1 advertises 10.244.2.0/24 via 10.112.0.52
! upstream switch learns the pod subnet and forwards natively

Its policy model is the richest of the three: standard NetworkPolicy plus Calico-specific policies with explicit allow/deny/log ordering, and iptables or eBPF data-plane options. If a cluster must integrate with the data-centre underlay, Calico in BGP mode is the natural fit - and the underlay side is ordinary L3 fabric design, with VTEP-style alternatives described in Linux VXLAN VTEP configuration.

Calico's own extended policy is where the extra control lives. A standard Kubernetes NetworkPolicy cannot express "allow this egress to a specific CIDR, deny everything else, and log the denies" in one object; a Calico GlobalNetworkPolicy can, with an explicit order and an action per rule:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-dns-egress
spec:
  order: 100
  selector: all()
  egress:
    - action: Allow
      protocol: UDP
      destination:
        ports: [53]
    - action: Log
    - action: Deny

On a routed fabric the pod MTU problem largely disappears - there is no outer header - which is one reason Calico in BGP mode measures close to line rate.

Cilium - eBPF Everywhere

Cilium replaces kube-proxy with eBPF programs, supports native routing, GENEVE tunnelling or WireGuard/IPsec encryption, and enforces policy from L3 up to L7 HTTP rules without sidecars. It is the strongest default for a new production cluster in 2026, provided the kernel is modern (5.6+ for the encryption modes) and the team is comfortable debugging with cilium status and Hubble rather than iptables counters.

The kube-proxy replacement matters more than it sounds. In a traditional iptables cluster, Service DNAT is a chain of iptables rules that is walked per packet and that grows with the number of Services and endpoints. At a few hundred Services that is invisible; at a few thousand it becomes measurable latency and a real CPU cost on every node. eBPF replaces that linear chain with a hash-map lookup in the kernel, so Service performance stays roughly flat as the cluster grows. The same machinery gives Cilium its L7 policy - it can allow HTTP GET to /api/v1/health and drop POST, without an application-level sidecar:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-api
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: ^/api/v1/health$

Cilium's cost is that it moves the debugging interface. When something does not work, you do not read iptables -L; you read cilium status, cilium monitor and Hubble flows. That is a genuine skill investment, not a config flag.

Choosing

• Flannel - labs, edge k3s clusters, or any case where pure connectivity with zero policy is genuinely the requirement.
• Calico - enterprise clusters needing strong policy, BGP integration with existing routers and predictable performance without encapsulation.
• Cilium - performance-sensitive, security-focused platforms that want kube-proxy removal, transparent encryption and L7 visibility.

Installing Them (and Not Getting Stuck)

The install itself is a single manifest apply, but two pitfalls cause most first-cluster failures: an incompatible pod CIDR and a busy host port. Calico's default pod CIDR is 192.168.0.0/16 and Cilium's is 10.0.0.0/8; if either collides with your node network, pods will come up and be unreachable. Set it explicitly at install time:

# Calico
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/tigera-operator.yaml
# then set spec.calicoNetwork.ipPools[0].cidr in the custom resource

# Cilium (Helm), explicit pod CIDR and kube-proxy replacement
helm install cilium cilium/cilium --namespace kube-system \
  --set ipam.operator.clusterPoolIPv4PodCIDRList=10.244.0.0/16 \
  --set kubeProxyReplacement=true \
  --set routingMode=native

Verify before you build anything on top:

kubectl -n kube-system get pods -o wide
kubectl get nodes
cilium status                    # Cilium
calicoctl node status            # Calico
kubectl run nettest --image=nicolaka/netshoot -- sleep 3600
kubectl exec nettest -- ping -c3 10.244.2.5   # another node's pod

Overlay vs Routed: The Cost in Numbers

The performance gap between the three plugins is not a marketing claim; it is arithmetic. An overlay transport wraps every pod packet in an extra outer header before it hits the wire, and that header consumes header-processing cycles on every hop and reduces the payload that fits in a given MTU. A routed data plane (Calico in BGP mode, Cilium in native routing) sends the pod packet essentially as-is, so the same physical link carries more application payload and the kernel does less work per packet.

At a small scale the difference is academic - a two-node lab will not notice 50 bytes of overhead. At scale it concentrates: a cluster running a service mesh, a storage network and high-throughput east-west traffic will hit the encapsulation ceiling first. The measurements most often cited, and reproduced by anyone who runs iperf3 between pods on a 10G fabric, are in the region of 6-7 Gbps for VXLAN-flavoured overlays against 9-10 Gbps for routed or native-eBPF options. The exact figures depend on packet size, kernel version, NIC offloads and whether the traffic crosses a node boundary, but the direction of the gap is consistent.

The corollary is that the encapsulation also changes the MTU you must configure. Whatever the physical MTU is, the pod interface must be that value minus the overlay header, or full-size packets will be dropped or fragmented. This single detail causes more "works for ping, hangs on large transfers" tickets than any other Kubernetes networking issue.

Policy Depth Compared

Connectivity is table stakes; policy is where the three diverge most sharply, so it is worth stating the capability boundaries plainly rather than in adjectives.

• Flannel has no policy engine at all. Multi-tenancy requires a second CNI plugin layered on top, which means two controllers, two failure domains and two upgrade paths for one requirement.
• Calico supports the Kubernetes NetworkPolicy API plus its own extended policies. The extension matters because standard NetworkPolicy has no notion of rule ordering, no explicit deny, and no logging - a Calico GlobalNetworkPolicy adds all three, which is what makes it usable for compliance-driven isolation.
• Cilium goes further up the stack: the same policy object can match L7 HTTP methods and paths, DNS names, and Kafka or gRPC messages, all enforced in eBPF without a sidecar. That is a capability neither of the others offers, and it is the reason Cilium is common in security-sensitive platforms.

The decision rule that follows: if you need only connectivity, the simple overlay wins on operational cost. If you need strong, auditable L3/L4 policy that integrates with your existing routers, Calico's routed mode is the natural fit. If you need L7 policy, transparent encryption and kube-proxy removal, Cilium is the only one of the three that delivers them natively.

Troubleshooting the Data Plane

• Pods up, no cross-node connectivity: check the pod CIDR collision first, then the host firewall - a default-deny INPUT chain on the node will drop VXLAN (UDP 8472 or 4789) or IPIP (protocol 4).
• Large transfers hang, small requests work: an MTU mismatch. Confirm the pod interface MTU equals physical minus the encapsulation overhead.
• Policy silently blocking: Calico logs denies in the calico-node logs and with a Log action; Cilium shows them as drops in cilium monitor --type drop and in Hubble.
• Service unreachable but pod-to-pod fine: kube-proxy or its eBPF replacement. In Cilium, confirm kubeProxyReplacement and that the cilium-agent is healthy on every node.

Whichever you pick, automate the day-2 work: address planning in NetBox IPAM, validation with pyATS/Genie state diffs where the pods talk to network devices, and fleet changes through Ansible network modules.

原文链接：https://dev.to/pendelabhargavasai/kubernetes-cni-complete-guide-flannel-vs-cilium-vs-calico-cloud-provider-cnis-5c6c