# Prometheus Operator: ServiceMonitor, PrometheusRule and the path to an alert

> How the Prometheus Operator's objects select each other by label: a Prometheus picks ServiceMonitors and PrometheusRules, a ServiceMonitor picks a Service and a named port, and a firing alert goes through Alertmanager to Slack.

Interactive graph: https://k8wiz.com/graphs/prometheus-operator-servicemonitor-alerting

With the Prometheus Operator you don't write a scrape configuration. You write small objects, and the operator builds the configuration from the ones a `Prometheus` object selects. Every hop is a label selector, and a selector that matches nothing is not an error. It is a target that never shows up.

This graph follows one application's metrics from its `/metrics` endpoint to a message in Slack, and marks each selector on the way.

## Resources

### Prometheus k8s

comes with an add-on, monitoring.coreos.com/v1, namespace monitoring

Describes one Prometheus deployment. Its selectors decide which ServiceMonitors and PrometheusRules it loads, from which namespaces. The kube-prometheus-stack chart sets them to `release: <release name>` by default.

### StatefulSet prometheus-k8s

created by a controller, apps/v1, namespace monitoring

Created by the operator from the Prometheus object. Each Pod runs Prometheus and a config reloader that loads a new configuration without a restart.

### Secret prometheus-k8s

created by a controller, v1, namespace monitoring

The generated `prometheus.yaml.gz`. The operator rewrites it whenever a selected ServiceMonitor or PodMonitor changes. Decompress it to see what a ServiceMonitor turned into.

### ServiceMonitor checkout

you write it, monitoring.coreos.com/v1, namespace shop

Says which Services to scrape and how. Its own labels must match the Prometheus's `serviceMonitorSelector`, its `selector` must match the Service's labels, and `port` is the name of a Service port.

### Service checkout

you write it, v1, namespace shop

Carries the label the ServiceMonitor selects and a port named `metrics`. Prometheus doesn't send scrapes through the Service: it uses it to find the Pods behind it.

### Pod ×3 checkout-…

created by a controller, v1, namespace shop

The scrape targets. Each Pod IP on port 9090 becomes one target, labelled with its namespace, Service and Pod name.

### PrometheusRule checkout-alerts

you write it, monitoring.coreos.com/v1, namespace shop

Alerting and recording rules. Selected by label like a ServiceMonitor. The operator validates the expressions on admission and mounts the rules into the Prometheus Pods.

### Alertmanager main

comes with an add-on, monitoring.coreos.com/v1, namespace monitoring

Describes the Alertmanager cluster and which AlertmanagerConfigs it merges into its routing tree. It groups alerts, silences them and sends the notifications.

### AlertmanagerConfig shop-slack

you write it, monitoring.coreos.com/v1alpha1, namespace shop

A team's own route and receiver. The operator adds a matcher for the object's namespace, so it only ever handles alerts whose `namespace` label is `shop`.

### Slack channel #shop-alerts

outside the cluster

Where the page lands, once the alert has fired for 10 minutes and passed the route's matchers.

## How the resources connect

- Prometheus k8s → StatefulSet prometheus-k8s: operator creates (creates)
- Prometheus k8s → Secret prometheus-k8s: operator generates (creates)
- StatefulSet prometheus-k8s → Secret prometheus-k8s: mounts (references)
- Prometheus k8s → ServiceMonitor checkout: serviceMonitorSelector (selects)
- Prometheus k8s → PrometheusRule checkout-alerts: ruleSelector (selects)
- ServiceMonitor checkout → Service checkout: selector + port name (selects)
- Service checkout → Pod ×3 checkout-…: selector (selects)
- StatefulSet prometheus-k8s → Pod ×3 checkout-…: scrapes /metrics (traffic)
- StatefulSet prometheus-k8s → Alertmanager main: sends firing alerts (traffic)
- Alertmanager main → AlertmanagerConfig shop-slack: alertmanagerConfigSelector (selects)
- Alertmanager main → Slack channel #shop-alerts: notifies (traffic)

## How it works

1. **The operator runs Prometheus.** From the Prometheus object the operator creates a StatefulSet, a governing Service and a Secret for the configuration. You never edit those; you edit the object.
2. **Prometheus selects the ServiceMonitor.** The operator lists ServiceMonitors in the namespaces `serviceMonitorNamespaceSelector` allows and keeps those whose labels match `serviceMonitorSelector`. `checkout` has `release: kube-prometheus-stack`, so it is in.
3. **The ServiceMonitor selects the Service.** Its `selector` matches the Service's labels, in the ServiceMonitor's own namespace unless `namespaceSelector` says otherwise. `port: metrics` must be the name of a port on that Service.
4. **Prometheus scrapes the Pods.** The generated job discovers the Service's endpoints and scrapes each Pod IP every 30 seconds. The targets appear under Status, Targets in the Prometheus UI, or at `/api/v1/targets`.
5. **A rule fires.** Prometheus evaluates the PrometheusRule's expression. When it has been true for 10 minutes, the alert moves from pending to firing and is sent to Alertmanager with its labels.
6. **Alertmanager routes it.** The alert's `namespace` is `shop` and its severity is `page`, so the route from `shop-slack` matches and the receiver posts to the channel.

## Common mistakes

- A ServiceMonitor without the label the Prometheus selects is ignored, with no event anywhere. Compare `kubectl get prometheus -o yaml` with the ServiceMonitor's labels first.
- `port` is a port name, not a number. An unnamed Service port can't be referenced; `targetPort` exists for that and is rarely what you want.
- The ServiceMonitor selects Services by the Service's labels, not by the Pod labels in its selector. A Service with no labels of its own is never matched.
- Prometheus needs RBAC to list Services, Endpoints and Pods in the target namespace. Without it the job exists and has no targets.
- An AlertmanagerConfig only sees alerts with its own namespace in the `namespace` label. An alert whose expression drops that label never reaches the team's route.

## The manifests you write

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: shop
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: checkout
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: shop
  labels:
    app: checkout
spec:
  selector:
    app: checkout
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-alerts
  namespace: shop
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: checkout
      rules:
        - alert: CheckoutErrorRateHigh
          expr: |
            sum(rate(http_requests_total{namespace="shop",service="checkout",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{namespace="shop",service="checkout"}[5m])) > 0.05
          for: 10m
          labels:
            severity: page
            team: shop
          annotations:
            summary: More than 5% of checkout requests fail
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: shop-slack
  namespace: shop
  labels:
    alertmanager: main
spec:
  route:
    receiver: slack
    groupBy: ["alertname"]
    matchers:
      - name: severity
        value: page
  receivers:
    - name: slack
      slackConfigs:
        - channel: "#shop-alerts"
          apiURL:
            name: slack-webhook
            key: url
```