k8wiz
All graphs

Kubernetes graph

Prometheus Operator: ServiceMonitor, PrometheusRule and the path to an alert

How the Prometheus Operator's objects select each other by label: a Prometheus picks ServiceMonitors and PrometheusRules, a ServiceMonitor picks a Service and a named port, and a firing alert goes through Alertmanager to Slack.

With the Prometheus Operator you don't write a scrape configuration. You write small objects, and the operator builds the configuration from the ones a Prometheus object selects. Every hop is a label selector, and a selector that matches nothing is not an error. It is a target that never shows up.

This graph follows one application's metrics from its /metrics endpoint to a message in Slack, and marks each selector on the way.

Interactive graph

Click a resource to see what it does, or follow the setup step by step.

Comes with an add-onCreated by a controllerYou write itOutside the clustercreatesreferencesselects by labeltraffic

How it works

All 10 resources and what they do

Prometheus k8smonitoring.coreos.com/v1
Describes one Prometheus deployment. Its selectors decide which ServiceMonitors and PrometheusRules it loads, from which namespaces. The kube-prometheus-stack chart sets them to release: <release name> by default.
StatefulSet prometheus-k8sapps/v1
Created by the operator from the Prometheus object. Each Pod runs Prometheus and a config reloader that loads a new configuration without a restart.
Secret prometheus-k8sv1
The generated prometheus.yaml.gz. The operator rewrites it whenever a selected ServiceMonitor or PodMonitor changes. Decompress it to see what a ServiceMonitor turned into.
ServiceMonitor checkoutmonitoring.coreos.com/v1
Says which Services to scrape and how. Its own labels must match the Prometheus's serviceMonitorSelector, its selector must match the Service's labels, and port is the name of a Service port.
Service checkoutv1
Carries the label the ServiceMonitor selects and a port named metrics. Prometheus doesn't send scrapes through the Service: it uses it to find the Pods behind it.
Pod checkout-…v1
The scrape targets. Each Pod IP on port 9090 becomes one target, labelled with its namespace, Service and Pod name.
PrometheusRule checkout-alertsmonitoring.coreos.com/v1
Alerting and recording rules. Selected by label like a ServiceMonitor. The operator validates the expressions on admission and mounts the rules into the Prometheus Pods.
Alertmanager mainmonitoring.coreos.com/v1
Describes the Alertmanager cluster and which AlertmanagerConfigs it merges into its routing tree. It groups alerts, silences them and sends the notifications.
AlertmanagerConfig shop-slackmonitoring.coreos.com/v1alpha1
A team's own route and receiver. The operator adds a matcher for the object's namespace, so it only ever handles alerts whose namespace label is shop.
Slack channel #shop-alerts
Where the page lands, once the alert has fired for 10 minutes and passed the route's matchers.

Common mistakes

  • A ServiceMonitor without the label the Prometheus selects is ignored, with no event anywhere. Compare kubectl get prometheus -o yaml with the ServiceMonitor's labels first.
  • port is a port name, not a number. An unnamed Service port can't be referenced; targetPort exists for that and is rarely what you want.
  • The ServiceMonitor selects Services by the Service's labels, not by the Pod labels in its selector. A Service with no labels of its own is never matched.
  • Prometheus needs RBAC to list Services, Endpoints and Pods in the target namespace. Without it the job exists and has no targets.
  • An AlertmanagerConfig only sees alerts with its own namespace in the namespace label. An alert whose expression drops that label never reaches the team's route.

The manifests you write

Everything above that is marked "You write it", in one file. The rest comes from add-ons and controllers.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: shop
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: checkout
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: shop
  labels:
    app: checkout
spec:
  selector:
    app: checkout
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-alerts
  namespace: shop
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: checkout
      rules:
        - alert: CheckoutErrorRateHigh
          expr: |
            sum(rate(http_requests_total{namespace="shop",service="checkout",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{namespace="shop",service="checkout"}[5m])) > 0.05
          for: 10m
          labels:
            severity: page
            team: shop
          annotations:
            summary: More than 5% of checkout requests fail
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: shop-slack
  namespace: shop
  labels:
    alertmanager: main
spec:
  route:
    receiver: slack
    groupBy: ["alertname"]
    matchers:
      - name: severity
        value: page
  receivers:
    - name: slack
      slackConfigs:
        - channel: "#shop-alerts"
          apiURL:
            name: slack-webhook
            key: url

Keep going

Kubernetes quizTest yourself on Pods, Deployments, Services, scheduling, storage, networking, security and kubectl.ArticleHow to debug OOMKilled containers in KubernetesArticleCVE-2026-123158 (*sshole): the OpenSSH password bypass on AlpineArticleHow to drain Kubernetes nodes safely with PodDisruptionBudgets

More graphs