apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: payments-egress
namespace: shop
spec:
endpointSelector:
matchLabels:
app: payments
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s:k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toFQDNs:
- matchName: api.stripe.com
toPorts:
- ports:
- port: "443"
protocol: TCP
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: finance
app: ledger
toPorts:
- ports:
- port: "8080"
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ledger
namespace: finance
spec:
selector:
app: ledger
ports:
- port: 8080Cilium graph
Cilium egress to a domain: CiliumNetworkPolicy, the DNS proxy and toFQDNs
How a CiliumNetworkPolicy lets a Pod reach one external domain: the DNS rule that sends lookups through Cilium's proxy, the toFQDNs rule that turns answers into allowed addresses, identities, and Hubble showing each verdict.
A NetworkPolicy allows addresses, and the addresses of an outside API change. Cilium's toFQDNs lets you write the name instead. Cilium learns which addresses the name has by watching the Pod's own DNS lookups, and allows only those.
The policy below lets payments reach api.stripe.com on port 443 and the ledger Service in another namespace, and nothing else.
Interactive graph
Click a resource to see what it does, or follow the setup step by step.
How it works
All 10 resources and what they do
- Pod payments-6d9f7-…v1
- The source of the traffic. The policy selects it by its
app: paymentslabel. - CiliumEndpoint payments-6d9f7-…cilium.io/v2
- Created by the agent for each Pod it manages. It holds the Pod's addresses, its numeric identity and whether ingress and egress policy are enforced for it.
- CiliumIdentity 48213cilium.io/v2
- A number that stands for one set of labels. Every Pod with the same labels shares it, and the datapath matches packets on this number, not on Pod IPs.
- CiliumNetworkPolicy payments-egresscilium.io/v2
- Three egress rules. The first allows DNS to CoreDNS and, through
rules.dns, sends it via the DNS proxy. The second allows port 443 to whateverapi.stripe.comresolved to. The third allows the ledger. Everything else from the Pod is dropped. - DaemonSet ciliumapps/v1
- The agent on each node. It loads the eBPF programs that enforce policy, runs the DNS proxy and keeps the cache of names and addresses that
toFQDNsrules read. - Deployment corednsapps/v1
- Answers the lookup. The Pod still addresses CoreDNS; the agent's proxy sits in the path, reads the question and the answer, and passes both on.
- External API api.stripe.com:443
- The one site the Pod may reach. Its addresses come and go, which is why the policy names it.
- Any other site example.org:443
- Resolves fine, since the DNS rule allows every name. The connection is dropped, because no rule allows the address it resolved to.
- Service ledgerv1
- An internal dependency in another namespace. Once a policy has egress rules, it needs its own
toEndpointsrule or it is dropped with the rest. - Deployment hubble-relayapps/v1
- Collects the flows each agent records.
hubble observe --verdict DROPPEDshows which connection was denied, from which Pod, to which name and port.
Common mistakes
- Without
rules.dnson the port 53 rule, DNS works and the proxy never sees it.toFQDNslearns nothing and every connection to the domain is dropped. matchNameis exact:stripe.comdoesn't coverapi.stripe.com.matchPattern: "*.stripe.com"covers subdomains and not the bare domain.- The Pod can only reach addresses it looked up itself while the policy was in place. An application that caches an address across a restart of the agent can find it dropped until it resolves again.
- A rule with
toFQDNscan't also holdtoEndpointsortoCIDR. Put them in separate rules of the sameegresslist. - Narrowing
rules.dnsto one name makes every other lookup fail with REFUSED, in-cluster names included. Keep a pattern for*.cluster.localor allow*.
The manifests you write
Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.
Keep going
More graphs
- Deployment, Service and Ingress: how a request reaches your Pods
- Gateway API: GatewayClass, Gateway and HTTPRoute with a canary split
- PVC, StorageClass and CSI: how a cloud disk ends up in your Pod
- RBAC: how a ServiceAccount gets its permissions
- cert-manager with Let's Encrypt: Certificate, CertificateRequest, Order and Challenge
- A private CA with cert-manager and trust-manager
- Istio traffic management: Gateway, VirtualService, DestinationRule and policies
- Flux dependencies: Git and OCI sources, Kustomizations and HelmReleases
- Argo CD: ApplicationSet, AppProject and the Applications it generates
- Inside an OCI image: tag, index, manifests, config and layers
- OCI artifacts: Helm charts, Flux artifacts and image volumes
- A Helm release: chart, values, revisions and the objects Helm owns
- Kustomize: a base, two overlays, a component and generated ConfigMaps
- A StatefulSet database: headless Service, per-Pod volumes, snapshots and restore
- Autoscaling end to end: KEDA, the HPA it creates and Karpenter adding nodes
- Prometheus Operator: ServiceMonitor, PrometheusRule and the path to an alert
- External Secrets Operator: from a cloud secret manager to a Pod, with rotation
- Kyverno CEL policies: ValidatingPolicy, MutatingPolicy, GeneratingPolicy and exceptions
- Istio ambient mode: ztunnel, waypoint proxies and authorization without sidecars
- Argo Rollouts canary: traffic weights on an HTTPRoute and analysis from Prometheus