Repair a broken rollout and keep its data

Operating a clusterAbout 60 minDraft

Not started.

What you need

Windows 11 with Docker Desktop running, and either PowerShell or a WSL terminal. The verifier runs inside a throwaway Node container, so nothing else needs installing.

Steps

  1. Save app.yaml and verify.mjs into an empty folder. Start a one-node Kubernetes with the command under "Start the cluster", in that folder. It takes about a minute; kubectl get nodes (the kubectl command below, then get nodes) shows it Ready.
  2. Apply the manifest: kubectl apply -f /work/app.yaml. Run the verifier: it fails. Use kubectl get pods, kubectl describe pod -l app=web and kubectl get endpointslices to find out why no pod is ready and why the Service routes nowhere.
  3. Fix both faults in app.yaml, and add a PersistentVolumeClaim mounted at /data in the web container. Apply it again and run the verifier until it passes; it writes a check file into /data.
  4. Roll out a broken version: kubectl set image deployment/web web=nginx:1.27-alpine-missing, then kubectl rollout status deployment/web --timeout=30s. It stalls. Notice the old pods keep serving.
  5. Roll back: kubectl rollout undo deployment/web, and wait for rollout status to finish.
  6. Run the verifier with --after at the end: two new revisions, the old image back, and the check file still there. Paste its last line below.

Why it matters

A readiness probe decides whether a pod gets traffic. When it asks the wrong port, the pod runs but never becomes ready, and the events on the pod say so. A Service finds pods only by label, so one wrong letter in its selector leaves it with no endpoints and no error at all.

The filesystem inside a pod goes with the pod. A PersistentVolumeClaim asks the cluster for storage that outlives any pod; k3s fills claims from a directory on its node, so here the data lives as long as the cluster container.

A rolling update replaces pods a few at a time and only moves on when new ones are ready. With a bad image the new pods never become ready, so the rollout stalls and the old pods keep serving: the readiness probe is what made that safe. rollout undo goes back to the previous template, as a new revision.

When you are done

  • docker rm -f -v skilltree-k3s
  • Delete the folder, including .skilltree-k8s.json

Files

Show app.yaml
# Starting point for the k8s-rollout task. It applies cleanly, but no pod ever becomes ready and the Service sends
# traffic nowhere. Find both faults from what the cluster tells you, fix them, and give the pods a /data directory
# on a PersistentVolumeClaim, so data written there outlives any single pod.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: website
  ports:
    - port: 80
      targetPort: 80

It asks Docker for what it needs and changes nothing else; read it before you run it. Mounting the Docker socket gives it the same control over Docker that you have.

Start the cluster

In the folder with your files, in PowerShell or in WSL:

docker run -d --name skilltree-k3s --privileged -v "${PWD}:/work" rancher/k3s:v1.33.5-k3s1 server --disable traefik --disable metrics-server

Run kubectl

Followed by the kubectl command; your folder is /work inside the cluster:

docker exec skilltree-k3s kubectl

Run the verifier

In the folder with your files, in PowerShell or in WSL:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "${PWD}:/work" -w /work node:22-alpine node verify.mjs k8s-rollout

The last step adds --after to the end.

The verifier runs on your machine, so this page cannot see the result for itself: what you paste is what it records.

Checks

First run

  • The cluster is running and its node is Readynot yet checked
  • Every web pod is readynot yet checked
  • The container has a readiness probenot yet checked
  • The Service routes to the ready podsnot yet checked
  • /data is a volume OpenTofu managesnot yet checked
  • A check file was writtennot yet checked

The final run (--after)

  • The cluster is running and its node is Readynot yet checked
  • Every web pod is readynot yet checked
  • The container has a readiness probenot yet checked
  • The Service routes to the ready podsnot yet checked
  • /data is a volume OpenTofu managesnot yet checked
  • The first run was recordednot yet checked
  • A bad rollout and a rollback happenednot yet checked
  • The previous image is backnot yet checked
  • The check file survivednot yet checked
Esc
↑ ↓ to move↵ to open