On a cloud provider, a Service of type LoadBalancer gets an external IP within a minute. On bare metal or in a lab cluster, the same Service sits at <pending> forever, because nothing in the cluster knows how to hand out an address or make the network deliver traffic to it.
Many teams install MetalLB to fill that gap. If you already run Cilium, you don't need a second component. Cilium has two features that together do the same job. LB IPAM allocates addresses from pools you define, and L2 announcements make one node answer ARP requests for each address so the rest of the LAN can reach it. This guide sets both up on Cilium 1.20, explains how failover works and shows how to debug the usual failures. L2 announcements is still marked beta in the Cilium docs, so test it before you depend on it.
How the two pieces fit together
LB IPAM only does bookkeeping. It watches Services of type LoadBalancer, picks a free address from a matching CiliumLoadBalancerIPPool and writes it into the Service's .status.loadBalancer.ingress. It is always present but dormant, and it wakes up when you create the first pool. On its own it makes the IP show up in kubectl get svc, but no packet will ever arrive.
L2 announcements does the delivery. For every selected Service, the Cilium agents on eligible nodes run a leader election through a Kubernetes Lease. The winner answers ARP requests (NDP for IPv6) for the Service IP with its own MAC address. Clients on the same layer 2 segment send traffic to that node, and Cilium's service load balancing forwards it to the backend pods, wherever they run.
This is the same model as a classic VRRP pair or MetalLB in layer 2 mode. One node takes all inbound traffic for a given IP, so you get failover but not load spreading across nodes before the cluster. If you need traffic spread across nodes, or your network team runs BGP, Cilium's BGP control plane is the better tool. For an office network, a homelab or a rack with a flat VLAN, L2 is simpler.
Step 1: check the prerequisites
L2 announcements requires Cilium's kube-proxy replacement. The announcing node hands traffic to Cilium's own eBPF service load balancer, so the feature doesn't work when kube-proxy handles Services. Check the current setting from any agent pod.
kubectl -n kube-system exec ds/cilium -- cilium-dbg config --all | grep KubeProxyReplacement
KubeProxyReplacement : true
If it says false, you have a bigger migration ahead, since switching to kube-proxy replacement means removing kube-proxy and pointing Cilium at the API server directly. Do that first and come back.
You also need a range of free addresses on the node network. Pick addresses that your DHCP server won't hand out and that no other device uses. In this guide the nodes sit on 192.168.10.0/24 and the range 192.168.10.200 to 192.168.10.230 is reserved for Services.
Finally, the network interface that carries this traffic must be one Cilium manages. If you set the devices Helm value explicitly, the interface has to be in that list. If you left devices empty, Cilium auto-detects the interfaces that have addresses, which usually includes the main NIC.
Step 2: enable L2 announcements and raise the client rate limit
Leader election is not free. Every node that is a candidate for a Service keeps polling its Lease, and the holder renews it every few seconds. That means steady traffic to the API server from each Cilium agent, and the agent's client rate limit can become the bottleneck.
The Cilium docs give a worst-case formula for the agent's request rate.
QPS = number of announced services / leaseRenewDeadline
With the default renew deadline of 5 seconds, 40 announced Services cost about 8 requests per second. The chart's default agent limit in 1.20 is 10 QPS with a burst of 20, which is tight once you add the agent's other API traffic. With 65 Services and a 2 second deadline, you'd need around 32.5 QPS. Set the limit a little above your calculated value and the burst somewhat higher.
Enable the feature with Helm. Replace the API server address and port with your own (they are the values you already use for kube-proxy replacement).
helm upgrade cilium cilium/cilium \
--version 1.20.4 \
--namespace kube-system \
--reuse-values \
--set kubeProxyReplacement=true \
--set k8sServiceHost=192.168.10.10 \
--set k8sServicePort=6443 \
--set l2announcements.enabled=true \
--set k8sClientRateLimit.qps=30 \
--set k8sClientRateLimit.burst=50
The chart doesn't restart the agents when the ConfigMap changes (rollOutCiliumPods defaults to false), so restart them yourself, along with the operator.
kubectl -n kube-system rollout restart ds/cilium deployment/cilium-operator
kubectl -n kube-system rollout status ds/cilium
kubectl -n kube-system exec ds/cilium -- cilium-dbg config --all | grep EnableL2Announcements
EnableL2Announcements : true
Using the Helm chart matters here. It also grants the agent's ClusterRole permission to create, get, update, list and delete leases in coordination.k8s.io. Clusters that enable the flag by editing the ConfigMap by hand often miss that, and the agent logs show a forbidden error on the lease.
Step 3: create an IP pool
The pool tells LB IPAM which addresses it may hand out. A block is either a CIDR or a start and stop pair. The range form is handy on a shared LAN, where you rarely own a clean CIDR.
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
name: lan-pool
spec:
blocks:
- start: "192.168.10.200"
stop: "192.168.10.230"
Watch the dashes. start and stop together form one block, so they must sit in a single list item. Two separate dashes create two broken blocks.
kubectl apply -f lan-pool.yaml
kubectl get ippools
The output shows whether the pool is disabled, whether it conflicts with another pool and how many addresses are still free. Check the CONFLICTING column every time you add a pool. Pools may not overlap, and when they do, the newest one is marked conflicting and stops allocating. That is a soft error, so nothing else warns you.
Two more pool options are worth knowing. With a CIDR block, LB IPAM hands out every address including the first and last; set spec.allowFirstLastIPs: "No" if your network treats those as network and broadcast addresses. And spec.serviceSelector restricts a pool to matching Services, which lets you give a tenant namespace its own range with the special label io.kubernetes.service.namespace.
Step 4: create an L2 announcement policy
Without a policy, nothing is announced. The policy picks which Services to announce, which nodes may announce them and on which interfaces.
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
name: lan-announce
spec:
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
interfaces:
- ^eth[0-9]+
loadBalancerIPs: true
externalIPs: false
The loadBalancerIPs and externalIPs fields both default to false. A policy that sets neither is valid, applies cleanly and announces nothing, which is the most common reason a first attempt fails silently. Set loadBalancerIPs: true to announce addresses assigned by LB IPAM.
The node selector keeps control plane nodes out of the election. The node that wins a Service's Lease carries all of its inbound traffic, so put it on workers sized for that. The interfaces field takes Go regular expressions, ORed together. Adjust the pattern to your NIC names, such as ^ens[0-9]+ or ^bond0$. Leaving the field out announces on every interface Cilium manages. The interface filter is not a security boundary; the Service still answers on other interfaces if someone points traffic at it.
There is no serviceSelector here, so every eligible Service is announced. A Service is eligible when its loadBalancerClass is unset or set to io.cilium/l2-announcer.
kubectl apply -f lan-announce.yaml
kubectl get l2announcement
Step 5: expose a Service and test it
Create a test deployment and a LoadBalancer Service.
kubectl create deployment echo --image=nginx --replicas=2
kubectl expose deployment echo --type=LoadBalancer --port=80
kubectl get svc echo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo LoadBalancer 10.96.41.17 192.168.10.200 80:31544/TCP 6s
Now check that a node took the Lease. The Lease lives in the Cilium namespace and is named cilium-l2announce-<namespace>-<service>.
kubectl -n kube-system get lease cilium-l2announce-default-echo
NAME HOLDER AGE
cilium-l2announce-default-echo worker-2 12s
From a machine on the same LAN, curl http://192.168.10.200 should return the nginx welcome page. Running ip neigh show 192.168.10.200 on that machine shows the MAC address of worker-2.
To pin an address, use the lbipam.cilium.io/ips annotation instead of the deprecated spec.loadBalancerIP field. To put several Services on one IP, give them the same lbipam.cilium.io/sharing-key annotation. They share an address as long as their ports don't clash.
Step 6: understand and tune failover
When the holder node dies, the other candidates notice that the Lease wasn't renewed within leaseDuration and race to take it. The first request the API server processes wins. The new holder sends a gratuitous ARP reply on its interfaces so clients update their ARP tables straight away.
Three Helm values control the timing. l2announcements.leaseDuration defaults to 15s, l2announcements.leaseRenewDeadline to 5s and l2announcements.leaseRetryPeriod to 2s. Failover takes between leaseDuration - leaseRenewDeadline and leaseDuration + leaseRenewDeadline, so 10 to 20 seconds with the defaults.
You can shorten it. With a 3s duration, 1s renew deadline and 200ms retry period, failover drops to 2 to 4 seconds. The cost is more API traffic, since the renew rate goes up fivefold and the QPS formula from Step 2 grows with it. Raise the rate limit at the same time.
helm upgrade cilium cilium/cilium \
--version 1.20.4 \
--namespace kube-system \
--reuse-values \
--set l2announcements.leaseDuration=3s \
--set l2announcements.leaseRenewDeadline=1s \
--set l2announcements.leaseRetryPeriod=200ms \
--set k8sClientRateLimit.qps=60 \
--set k8sClientRateLimit.burst=100
kubectl -n kube-system rollout restart ds/cilium
Gratuitous ARP has a catch. Some devices ignore unsolicited ARP replies because they can be used for spoofing. Those clients keep sending to the dead node until their ARP cache entry expires, so their outage lasts longer than the Lease settings suggest. Firewalls and some managed switches are the usual suspects.
Troubleshooting a Service that doesn't answer
Work from the top down. First, if EXTERNAL-IP stays <pending>, the problem is LB IPAM, not L2. The Service's status conditions say why.
kubectl get svc echo -o jsonpath='{.status.conditions}'
A no_pool reason means no enabled pool matches the Service. Check the pool's serviceSelector and the CONFLICTING column.
If the Service has an IP but no Lease exists, the policy doesn't select it, or the agent can't write Leases. Search the agent logs for forbidden (RBAC) or client-side throttling (rate limit too low). Throttling shows up as intermittent renew failures and frequent leader changes.
If the Lease exists, look at the holder's internal state. Find the Cilium pod on the holder node and dump its announcement table.
kubectl -n kube-system get pod -l app.kubernetes.io/name=cilium-agent -o wide --field-selector spec.nodeName=worker-2
kubectl -n kube-system exec cilium-7xk2p -- cilium-dbg shell -- db/show l2-announce
# IP NetworkInterface
192.168.10.200 eth0
A missing entry means the interface regex doesn't match a selected device. Run cilium-dbg shell -- db/show devices in the same pod and look at the Selected column. Only selected devices can announce.
If the table looks right but clients still can't connect, check externalTrafficPolicy. L2 announcements ignores the Local policy and may elect a node that has no backend pod. That node drops the traffic, because Local forbids forwarding to another node. Keep announced Services on externalTrafficPolicy: Cluster. Unless you run Cilium's DSR mode, that means backends see the forwarding node's address instead of the client's, which is the price of this design.