# Roll out strict mTLS in Istio without breaking traffic

> Move a namespace to STRICT mutual TLS. First find the client with no sidecar and the DestinationRule that turns TLS off, or their calls will fail.

Level: Advanced · Topic: Security · About 15 minutes
Try it in the browser: https://k8wiz.com/playground/roll-out-strict-mtls-in-istio-without-breaking-traffic

The playground is a simulated Kubernetes cluster that runs in the browser. kubectl output and errors follow the real ones.

## The situation

The `payments` namespace runs `api` inside the Istio mesh. Security wants it to accept mutual TLS only, with a `PeerAuthentication` in `STRICT` mode. Today it runs in the default `PERMISSIVE` mode and takes plain text too.

Two clients call `http://api.payments:8080`: `web` in `shop` and `nightly` in `reports`. Both work now. Applying STRICT as things stand would break both, each for a different reason.

Find the two reasons, fix them, then lock the namespace down. Test a client with `kubectl exec -n reports deploy/nightly -- curl -s -m 5 http://api.payments:8080`.

## Objectives

1. See which Pods have a sidecar and which namespaces inject one (`istioctl proxy-status` and `kubectl get namespaces -L istio-injection`)
2. nightly in reports runs with a sidecar
3. No DestinationRule turns client TLS off for api.payments (`kubectl get destinationrules -A -o yaml`)
4. payments accepts only mutual TLS: a Pod without a sidecar is refused (A namespace-wide PeerAuthentication named `default` with `mtls.mode: STRICT`)
5. web and nightly still reach api

## Hints

1. `istioctl proxy-status` lists every Pod with a sidecar: `api` and `web` are there, `nightly` isn't. `kubectl get namespaces -L istio-injection` shows `reports` was never labelled. A Pod without a sidecar sends plain text, which STRICT resets.
2. `kubectl label namespace reports istio-injection=enabled`, then `kubectl rollout restart deployment/nightly -n reports`. Injection happens when a Pod is created, so the restart is what adds the sidecar. The Pod then shows 2/2.
3. `web` has a sidecar, yet `kubectl get destinationrules -A` shows `api-legacy` in `shop` with `trafficPolicy.tls.mode: DISABLE`. It tells web's sidecar to send plain text to api, whatever auto mTLS would do. Delete it, or set the mode to `ISTIO_MUTUAL`.
4. Now lock it down: a PeerAuthentication named `default` in `payments`, no selector, `spec.mtls.mode: STRICT`. Test from a Pod outside the mesh: `kubectl run mtls-probe --image=curlimages/curl:8.16.0 -n default --restart=Never --command -- sleep infinity`, then `kubectl exec -n default mtls-probe -- curl -s -m 5 http://api.payments:8080` ends with exit code 56.
5. If a client breaks after the switch, set the mode back to `PERMISSIVE`. Istio pushes the change to the proxies within seconds.

## Solution

```bash
istioctl proxy-status
```
```bash
kubectl get namespaces -L istio-injection
```
```bash
kubectl label namespace reports istio-injection=enabled
```
```bash
kubectl rollout restart deployment/nightly -n reports
```
```bash
kubectl rollout status deployment/nightly -n reports
```
```bash
kubectl delete destinationrule api-legacy -n shop
```
```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT
EOF
```
```bash
kubectl wait --for=delete pod -n reports -l app=nightly,security.istio.io/tlsMode!=istio --timeout=60s
```
```bash
kubectl exec -n reports deploy/nightly -- curl -s -m 5 http://api.payments:8080
```
```bash
kubectl exec -n shop deploy/web -- curl -s -m 5 http://api.payments:8080
```

## What it teaches

- Mutual TLS in Istio has a client half and a server half. Auto mTLS makes a sidecar send mutual TLS to destinations that have a proxy; a `PeerAuthentication` decides what the server accepts.
- `PERMISSIVE` takes mutual TLS and plain text, `STRICT` only mutual TLS. The narrowest policy wins: workload, then namespace, then the mesh-wide one in `istio-system`.
- A client without a sidecar sends plain text. Under STRICT its connection is reset and curl exits with code 56. Label its namespace for injection and restart the workload before you lock down.
- A DestinationRule with `tls.mode: DISABLE` overrides auto mTLS for its host. The client then gets a 503 from its own sidecar once the server goes STRICT.
- Go namespace by namespace, and keep `PERMISSIVE` as the quick way back.