k8wiz
All graphs

Cilium graph

Cilium egress to a domain: CiliumNetworkPolicy, the DNS proxy and toFQDNs

How a CiliumNetworkPolicy lets a Pod reach one external domain: the DNS rule that sends lookups through Cilium's proxy, the toFQDNs rule that turns answers into allowed addresses, identities, and Hubble showing each verdict.

A NetworkPolicy allows addresses, and the addresses of an outside API change. Cilium's toFQDNs lets you write the name instead. Cilium learns which addresses the name has by watching the Pod's own DNS lookups, and allows only those.

The policy below lets payments reach api.stripe.com on port 443 and the ledger Service in another namespace, and nothing else.

Interactive graph

Click a resource to see what it does, or follow the setup step by step.

Created by a controllerYou write itComes with an add-onOutside the clusterselects by labelcreatesreferencestraffic

How it works

All 10 resources and what they do

Pod payments-6d9f7-…v1
The source of the traffic. The policy selects it by its app: payments label.
CiliumEndpoint payments-6d9f7-…cilium.io/v2
Created by the agent for each Pod it manages. It holds the Pod's addresses, its numeric identity and whether ingress and egress policy are enforced for it.
CiliumIdentity 48213cilium.io/v2
A number that stands for one set of labels. Every Pod with the same labels shares it, and the datapath matches packets on this number, not on Pod IPs.
CiliumNetworkPolicy payments-egresscilium.io/v2
Three egress rules. The first allows DNS to CoreDNS and, through rules.dns, sends it via the DNS proxy. The second allows port 443 to whatever api.stripe.com resolved to. The third allows the ledger. Everything else from the Pod is dropped.
DaemonSet ciliumapps/v1
The agent on each node. It loads the eBPF programs that enforce policy, runs the DNS proxy and keeps the cache of names and addresses that toFQDNs rules read.
Deployment corednsapps/v1
Answers the lookup. The Pod still addresses CoreDNS; the agent's proxy sits in the path, reads the question and the answer, and passes both on.
External API api.stripe.com:443
The one site the Pod may reach. Its addresses come and go, which is why the policy names it.
Any other site example.org:443
Resolves fine, since the DNS rule allows every name. The connection is dropped, because no rule allows the address it resolved to.
Service ledgerv1
An internal dependency in another namespace. Once a policy has egress rules, it needs its own toEndpoints rule or it is dropped with the rest.
Deployment hubble-relayapps/v1
Collects the flows each agent records. hubble observe --verdict DROPPED shows which connection was denied, from which Pod, to which name and port.

Common mistakes

  • Without rules.dns on the port 53 rule, DNS works and the proxy never sees it. toFQDNs learns nothing and every connection to the domain is dropped.
  • matchName is exact: stripe.com doesn't cover api.stripe.com. matchPattern: "*.stripe.com" covers subdomains and not the bare domain.
  • The Pod can only reach addresses it looked up itself while the policy was in place. An application that caches an address across a restart of the agent can find it dropped until it resolves again.
  • A rule with toFQDNs can't also hold toEndpoints or toCIDR. Put them in separate rules of the same egress list.
  • Narrowing rules.dns to one name makes every other lookup fail with REFUSED, in-cluster names included. Keep a pattern for *.cluster.local or allow *.

The manifests you write

Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-egress
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: payments
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: api.stripe.com
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: finance
            app: ledger
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: ledger
  namespace: finance
spec:
  selector:
    app: ledger
  ports:
    - port: 8080

Keep going

More graphs