MetalLB Layer2 vs BGP: Address Pool Setup

In a cloud cluster, type: LoadBalancer just works because the provider hands out an IP. On bare metal nothing does that for you — the service sits forever at EXTERNAL-IP: <pending>. MetalLB fills that gap: it owns a pool of addresses you control and advertises them to the network, either by answering ARP/ND for the whole pool (Layer2 mode) or by peering with your routers and advertising the addresses over BGP. Choosing the wrong mode is the difference between "works fine" and "all traffic hits one node".

Layer2 vs BGP in One Table

Install and Define the Pool

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl -n metallb-system get pods            # controller + speaker must be Running

Address pools are the modern API. Keep "cheap" and "expensive" ranges as separate pools, and disable automatic allocation on the expensive one so only services that ask for it get an address from it:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: cheap
  namespace: metallb-system
spec:
  addresses:
    - 192.168.10.0/24
  autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: expensive
  namespace: metallb-system
spec:
  addresses:
    - 42.176.25.64/30
  autoAssign: false          # only services that request this pool by name

A service selects a specific pool with an annotation:

apiVersion: v1
kind: Service
metadata:
  name: ingress-public
  annotations:
    metallb.universe.tf/address-pool: expensive
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - port: 443
      targetPort: 8443

Then tell MetalLB how to advertise it. Layer2 needs an L2Advertisement; BGP needs a BGPPeer plus a BGPAdvertisement:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-default
  namespace: metallb-system
spec:
  ipAddressPools:
    - cheap
---
apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
  name: rack-tor
  namespace: metallb-system
spec:
  myASN: 64512
  peerASN: 64500
  peerAddress: 192.168.10.1
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgp-default
  namespace: metallb-system
spec:
  ipAddressPools:
    - expensive

Verify That It Really Works

kubectl get svc                                # EXTERNAL-IP must leave <pending>
kubectl get ipaddresspool -n metallb-system
kubectl -n metallb-system logs deploy/metallb-controller --tail=50
kubectl -n metallb-system logs ds/metallb-speaker --tail=50   # speaker elects / announces

# from a client outside the cluster
curl -k https://42.176.25.64/
arping -I eth0 192.168.10.10                   # Layer2: which MAC answers?
ip route get 42.176.25.64                      # BGP: which next-hop is used?

How Layer2 Leader Election Actually Works

In Layer2 mode the speakers form a cluster and elect exactly one node to answer ARP (IPv4) or NDP (IPv6) for each service address. Only that node replies for the address; the others stay passive. When the leader fails, the remaining speakers re-elect and the new leader sends a gratuitous ARP so the network updates its MAC cache. The catch is that failover is only as fast as the surrounding switches and routers refresh their neighbour entries — the cluster may converge in milliseconds while a router holds the stale MAC for another thirty seconds.

Constraining Which Nodes Announce

An L2Advertisement can restrict which nodes may become leader, using nodeSelectors to limit by label and interfaces to limit by NIC name:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-rack1
  namespace: metallb-system
spec:
  ipAddressPools:
    - cheap
  nodeSelectors:
    - matchLabels:
        rack: rack1
  interfaces:
    - ens192

This matters on multi-rack clusters: if a pool belongs to rack1's layer-2 segment, only rack1 nodes can answer for it. Without the selector, MetalLB may elect a node whose ARP replies never reach the clients that use the address.

FRR Mode: Real ECMP

BGP mode without FRR still pins each service to a single node, which is why it can look like Layer2 with more configuration. Enable the FRR (Free Range Routing) backend and each node advertises the service address independently, so the upstream router sees several equal-cost next-hops and spreads connections across them. Combined with externalTrafficPolicy: Local, a client's connection lands on a node that actually runs a pod, and traffic is not re-hashed across the cluster. That combination is what makes BGP mode deliver the even distribution and throughput that Layer2 cannot.

BFD for Sub-Second Failover

Plain BGP waits for hold timers to expire before it withdraws a failed session, which can stretch failover into seconds. Turning on BFD (Bidirectional Forwarding Detection) for the peer shortens detection to sub-second by exchanging lightweight hello packets:

apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
  name: rack-tor
  namespace: metallb-system
spec:
  myASN: 64512
  peerASN: 64500
  peerAddress: 192.168.10.1
  bfdProfile: fast

BFD does not create paths, but it tells BGP about a dead neighbour quickly, so the router stops black-holing traffic sooner.

Dual-Stack and IPv6 Pools

A single pool can carry both IPv4 and IPv6 addresses; MetalLB advertises IPv6 prefixes over BGP and answers NDP in Layer2 mode just as it does ARP for IPv4. For a dual-stack service, request both families so clients reach it over either protocol:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: dualstack
  namespace: metallb-system
spec:
  addresses:
    - 192.168.10.0/24
    - 2001:db8:10::/64

Monitoring the Speakers

kubectl -n metallb-system get pods -o wide
kubectl -n metallb-system logs ds/metallb-speaker --tail=100 | grep -i "assigned\|elected\|announced"
kubectl -n metallb-system get events --field-selector involvedObject.kind=Service
kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.name}{"\t"}{.status.loadBalancer.ingress[0].ip}{"\n"}{end}'

The speaker log is the fastest way to see which node holds a Layer2 address and whether a BGP session flapped.

Migrating from Layer2 to BGP

You do not have to rebuild the cluster. Add a BGPPeer and BGPAdvertisement for a new (or the same) pool, confirm the sessions are Established, then move services one at a time by switching their address-pool annotation. While both advertisements exist you can leak the address from either side, which makes the cutover reversible. Test with a long-running connection before you retire the Layer2 advertisement — a live flow will break when the address changes how it is announced. The router side of that peering (ASN, templates, sessions) is covered in our MikroTik RouterOS 7 BGP template and connection guide.

Sizing the Pool and Reserving Addresses

Size the pool to the number of LoadBalancer services you will realistically run, not just the ones today, and keep the range out of any DHCP scope. For BGP, the upstream router must accept the prefix; a /32 or /128 host route per service is typical. Avoid pools that overlap DHCP leases or other statically assigned addresses, or you will chase a duplicate-ARP conflict that has nothing to do with MetalLB.

When a Service Stays at <pending>

A LoadBalancer service that never leaves <pending> almost always has one of four causes. First, the pool is exhausted — every address is already taken by another service. Second, the service requests a pool by annotation that does not exist or is not covered by any advertisement, so no speaker can claim it. Third, the MetalLB controller is not running or is crash-looping, which you confirm with kubectl -n metallb-system get pods. Fourth, a pool with autoAssign: false silently ignores services that never name it. Check the controller log first: it prints exactly why an allocation was refused.

Layer2 vs BGP in Practice: Stickiness and Throughput

The two modes fail differently under load. Layer2 forwards every inbound packet through the elected node, so that node's NIC becomes the ceiling for the address; with a handful of high-bandwidth services this is usually fine, but a busy ingress controller on one IP can saturate its host. BGP with FRR spreads connections across nodes, but a single TCP flow still follows one ECMP path, so per-flow hashing matters. If your clients open many short connections, BGP distributes them well; if one client holds a single long-lived connection, neither mode will spread that flow for you.

External Traffic Policy and Source IP Preservation

externalTrafficPolicy changes both distribution and source-IP visibility. With Cluster, the receiving node may forward to a pod on another node, and the client's source IP is masked behind SNAT. With Local, only nodes running the pod accept the traffic, the client source IP is preserved, and there is no extra intra-cluster hop — at the cost of uneven distribution when replicas are not on every node. Pair Local with BGP and FRR to get both real spreading and preserved source addresses.

Securing the BGP Session

A BGP session with your top-of-rack switch is a trust relationship: anyone who hijacks it can black-hole your service addresses. Set a session password (password in the BGPPeer), restrict the peering to a dedicated management VLAN, and let the router accept only the exact prefixes you intend to advertise. On the router side, apply an inbound prefix filter and a maximum-prefix limit so a misconfigured speaker cannot flood the routing table.

Node Failure, Draining and Maintenance

When a node is drained for maintenance, the speakers withdraw its advertisements; in Layer2 mode leadership moves, and in BGP mode the router simply stops using that next-hop. Because a drain is graceful, failover is cleaner than an unplanned crash: the address moves before the node disappears. Schedule drains for any node that currently owns a Layer2 address, and watch the speaker log to confirm the handover completed before you power the node off.

Pitfalls That Waste Afternoons

• Pool not on a shared L2 segment: in Layer2 mode the elected node's ARP replies must reach clients. A pool advertised from a VLAN nobody routes to is invisible.
• Traffic all lands on one node: that is Layer2 mode's design, not a bug. Fix it with BGP + FRR and externalTrafficPolicy: Local, or accept it for small workloads.
• Announcing an address the router will not accept: in BGP mode your upstream must permit the prefix. If it filters the host route, nothing arrives.
• Overlapping pools: two pools covering the same range produce nondeterministic allocation across restarts.
• Switch/router ARP-cache lag: Layer2 failover can take seconds even though the cluster moved in milliseconds — lower ARP timers on the adjacent L3 device or move to BGP.

Router-Side Peering: What the Switch Needs

A MetalLB BGP session only works if the router accepts it. On the router you need a neighbour entry with the speaker’s address, the correct remote AS, a password that matches the BGPPeer, and an inbound policy that permits exactly the host routes MetalLB will advertise. A minimal top-of-rack configuration looks like this:

! Cisco NX-OS top-of-rack switch
router bgp 64500
  router-id 192.168.10.1
  neighbor 192.168.10.11
    remote-as 64512
    password <shared-secret>
    address-family ipv4 unicast
      send-community standard
      maximum-prefix 64

Three details matter more than the syntax. First, maximum-prefix caps how much damage a buggy speaker can do to the routing table. Second, an inbound prefix filter (a route-map or prefix-list) is what stops a stray pool from leaking into your routing domain. Third, the session password must match on both ends character for character — a mismatch shows up as a session stuck in Active or Idle, never as a helpful error message. Verify from the router’s point of view rather than the speaker’s:

show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors 192.168.10.11 advertised-routes
show bgp ipv4 unicast 42.176.25.64/32

If the speaker says the session is Established while the router shows nothing, the peering is fine and the advertisement is being filtered — a policy problem, not a MetalLB problem.

Pinning an Address and Sharing One IP

Auto-allocation is convenient until something outside the cluster depends on a specific address: a DNS record, a firewall rule, a customer allow-list. MetalLB lets a service request an exact address instead of whatever the pool hands out first, and lets several services share one address when their ports do not overlap:

apiVersion: v1
kind: Service
metadata:
  name: dns-udp
  annotations:
    metallb.universe.tf/loadBalancerIPs: 192.168.10.53
    metallb.universe.tf/allow-shared-ip: dns53
spec:
  type: LoadBalancer
  ports:
    - name: dns-udp
      port: 53
      protocol: UDP
      targetPort: 53

Two services may share one IP only if their allow-shared-ip value is identical and their port/protocol sets do not collide — which is exactly how DNS over TCP and UDP, or two web services on different ports, end up on the same address. If a pinned address is already held by a service that did not opt into sharing, MetalLB leaves the request unsatisfied and says so in the controller log. Reserve pinned addresses by removing them from the autoAssign: true pool or by carving a small static sub-range out of it, so an automatic allocation cannot steal the address your documentation promises.

Metrics Worth Alerting On

MetalLB exposes Prometheus metrics from the controller and from every speaker, and a handful of them catch most real failures:

metallb_bgp_session_up                             # 0 means the peering is down
metallb_speaker_announced                          # 0 means a service is not announced
metallb_allocator_addresses_in_use_total           # how much of the pool is used
rate(metallb_layer2_requests_received_total[5m])   # Layer2 ARP/ND request rate

Alert on metallb_bgp_session_up == 0 for more than a minute, on a service that is allocated but never announced, and on pool utilisation crossing roughly eighty percent. Pool exhaustion is quiet: services simply sit at <pending> until a human notices, so a capacity alert is the only warning you get. The scrape targets are the speaker and controller pods, and the same endpoint is usually the fastest way to separate an advertisement problem from a pool problem.

相关阅读：Kubernetes NetworkPolicy 默认拒绝模式、Calico、Cilium 与 Flannel CNI 对比 以及 Proxmox VE VLAN-Aware 网桥与 LACP 聚合。

原文链接：MetalLB - Advanced AddressPool configuration