# Autoscaling end to end: KEDA, the HPA it creates and Karpenter adding nodes

> From queue depth to new machines: a KEDA ScaledObject reads Prometheus, creates an HPA that scales a Deployment from zero, and when the Pods don't fit, Karpenter's NodePool launches a NodeClaim and a node.

Interactive graph: https://k8wiz.com/graphs/keda-hpa-karpenter-autoscaling

Scaling a workload on a business metric takes two layers. KEDA turns an outside signal, here the number of jobs waiting in a queue, into a HorizontalPodAutoscaler. When the new Pods don't fit on the nodes there are, Karpenter reads what they ask for and launches a machine that fits.

Neither knows about the other. The Pending Pod is the handover.

## Resources

### Service prometheus-k8s

comes with an add-on, v1, namespace monitoring

Where KEDA sends its query. The answer is one number, the jobs waiting in the `render` queue.

### TriggerAuthentication prometheus-token

you write it, keda.sh/v1alpha1, namespace media

Tells KEDA where to find the credentials for the trigger, here a bearer token in a Secret, so they stay out of the ScaledObject.

### ScaledObject render-worker

you write it, keda.sh/v1alpha1, namespace media

Names the workload, the bounds and the trigger. One replica per 20 waiting jobs, from 0 to 40. KEDA handles 0 to 1 itself and leaves 1 to 40 to the HPA it creates.

### HorizontalPodAutoscaler keda-hpa-render-worker

created by a controller, autoscaling/v2, namespace media

Created and owned by KEDA. Its one metric is of type `External`, served by KEDA's metrics adapter. Edit the ScaledObject, never this object: KEDA overwrites it.

### Deployment render-worker

you write it, apps/v1, namespace media

The workers. Their requests are what the scheduler and Karpenter size nodes by. No `replicas` in the manifest, so applying it doesn't fight the autoscaler.

### Pod ×12 render-worker-…

created by a controller, v1, namespace media

Some fit on the nodes there are. The rest stay Pending with `0/5 nodes are available: 5 Insufficient cpu`, which is the event Karpenter acts on.

### NodePool general

you write it, karpenter.sh/v1

The rules for nodes Karpenter may launch: which instance families, zones and capacity types, a ceiling on total CPU, and when to remove or replace nodes.

### EC2NodeClass default

you write it, karpenter.k8s.aws/v1

The cloud side of a node: which image, which subnets and security groups (found by tag) and the IAM role the instance runs as.

### NodeClaim general-x7k2p

created by a controller, karpenter.sh/v1

One request for one machine, with the instance types that would fit the Pending Pods. Karpenter creates it, launches the instance and tracks it until the node registers.

### Node ip-10-0-42-17

created by a controller, v1

The instance once its kubelet has joined, labelled with its NodePool. The scheduler places the Pending Pods on it, usually within a minute of the scale-up.

## How the resources connect

- ScaledObject render-worker → Service prometheus-k8s: query every 30s (traffic)
- ScaledObject render-worker → TriggerAuthentication prometheus-token: authenticationRef (references)
- ScaledObject render-worker → HorizontalPodAutoscaler keda-hpa-render-worker: creates and owns (creates)
- ScaledObject render-worker → Deployment render-worker: scaleTargetRef, 0↔1 (references)
- HorizontalPodAutoscaler keda-hpa-render-worker → Deployment render-worker: scales 1↔40 (references)
- Deployment render-worker → Pod ×12 render-worker-…: ReplicaSet creates (creates)
- NodePool general → Pod ×12 render-worker-…: watches Pending (selects)
- NodePool general → EC2NodeClass default: nodeClassRef (references)
- NodePool general → NodeClaim general-x7k2p: creates (creates)
- NodeClaim general-x7k2p → Node ip-10-0-42-17: launches (creates)
- Pod ×12 render-worker-… → Node ip-10-0-42-17: scheduled onto (references)

## How it works

1. **KEDA polls the metric.** Every 30 seconds the KEDA operator runs the query. While the answer is below `activationThreshold`, the Deployment stays at zero replicas and costs nothing.
2. **Zero to one is KEDA's job.** An HPA can't scale from zero on its own metric. When the queue has a job, KEDA sets the Deployment to one replica, and from there the HPA takes over.
3. **The HPA does the rest.** With 240 jobs waiting and a threshold of 20, the HPA asks for 12 replicas. It reads the number from KEDA's metrics adapter as an external metric.
4. **Pods that don't fit stay Pending.** Each worker requests 2 CPUs. The scheduler places what fits and marks the rest unschedulable. Nothing else is needed to trigger Karpenter.
5. **Karpenter launches a node that fits.** Karpenter batches the Pending Pods, adds up their requests and constraints, and creates a NodeClaim for the cheapest instance types the NodePool allows. The EC2NodeClass supplies image, subnets and role.
6. **The node joins, then leaves again.** The instance registers, the Pods are scheduled, and the queue drains. After the cooldown KEDA scales back to zero, and two minutes after the node is empty Karpenter removes it.

## Common mistakes

- A Deployment with `replicas` in Git and a GitOps tool that enforces it resets the count on every sync. Leave the field out, or tell Argo CD or Flux to ignore it.
- Pods without requests don't make Karpenter add nodes: as far as it can tell they fit anywhere.
- `limits.cpu` on the NodePool is a hard ceiling. When it is reached, Pods stay Pending and the NodePool's events say why.
- Consolidation evicts Pods to pack nodes tighter. Long jobs need a PodDisruptionBudget or the `karpenter.sh/do-not-disrupt: "true"` annotation, and a grace period long enough to finish.
- Two HPAs on one Deployment fight. Don't add your own next to the one KEDA creates; add a second trigger to the ScaledObject.

## The manifests you write

```yaml
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: prometheus-token
  namespace: media
spec:
  secretTargetRef:
    - parameter: bearerToken
      name: prometheus-reader
      key: token
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: render-worker
  namespace: media
spec:
  scaleTargetRef:
    name: render-worker
  minReplicaCount: 0
  maxReplicaCount: 40
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc:9090
        query: sum(queue_jobs_waiting{queue="render"})
        threshold: "20"
        activationThreshold: "1"
        authModes: bearer
      authenticationRef:
        name: prometheus-token
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: render-worker
  namespace: media
spec:
  selector:
    matchLabels:
      app: render-worker
  template:
    metadata:
      labels:
        app: render-worker
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: worker
          image: ghcr.io/acme/render-worker:5.2.0
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      expireAfter: 720h
  limits:
    cpu: "400"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-prod
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod
```