Repair a broken rollout and keep its data
Not started.
What you need
Windows 11 with Docker Desktop running, and either PowerShell or a WSL terminal. The verifier runs inside a throwaway Node container, so nothing else needs installing.
Steps
- Save
app.yamlandverify.mjsinto an empty folder. Start a one-node Kubernetes with the command under "Start the cluster", in that folder. It takes about a minute;kubectl get nodes(the kubectl command below, thenget nodes) shows it Ready. - Apply the manifest:
kubectl apply -f /work/app.yaml. Run the verifier: it fails. Usekubectl get pods,kubectl describe pod -l app=webandkubectl get endpointslicesto find out why no pod is ready and why the Service routes nowhere. - Fix both faults in
app.yaml, and add a PersistentVolumeClaim mounted at /data in the web container. Apply it again and run the verifier until it passes; it writes a check file into /data. - Roll out a broken version:
kubectl set image deployment/web web=nginx:1.27-alpine-missing, thenkubectl rollout status deployment/web --timeout=30s. It stalls. Notice the old pods keep serving. - Roll back:
kubectl rollout undo deployment/web, and wait forrollout statusto finish. - Run the verifier with
--afterat the end: two new revisions, the old image back, and the check file still there. Paste its last line below.
Why it matters
A readiness probe decides whether a pod gets traffic. When it asks the wrong port, the pod runs but never becomes ready, and the events on the pod say so. A Service finds pods only by label, so one wrong letter in its selector leaves it with no endpoints and no error at all.
The filesystem inside a pod goes with the pod. A PersistentVolumeClaim asks the cluster for storage that outlives any pod; k3s fills claims from a directory on its node, so here the data lives as long as the cluster container.
A rolling update replaces pods a few at a time and only moves on when new ones are ready. With a bad image the new pods never become ready, so the rollout stalls and the old pods keep serving: the readiness probe is what made that safe. rollout undo goes back to the previous template, as a new revision.
When you are done
docker rm -f -v skilltree-k3sDelete the folder, including .skilltree-k8s.json
Files
Show app.yaml
# Starting point for the k8s-rollout task. It applies cleanly, but no pod ever becomes ready and the Service sends
# traffic nowhere. Find both faults from what the cluster tells you, fix them, and give the pods a /data directory
# on a PersistentVolumeClaim, so data written there outlives any single pod.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: nginx:1.27-alpine
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /
port: 8080
periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: website
ports:
- port: 80
targetPort: 80
It asks Docker for what it needs and changes nothing else; read it before you run it. Mounting the Docker socket gives it the same control over Docker that you have.
Start the cluster
In the folder with your files, in PowerShell or in WSL:
docker run -d --name skilltree-k3s --privileged -v "${PWD}:/work" rancher/k3s:v1.33.5-k3s1 server --disable traefik --disable metrics-serverRun kubectl
Followed by the kubectl command; your folder is /work inside the cluster:
docker exec skilltree-k3s kubectlRun the verifier
In the folder with your files, in PowerShell or in WSL:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "${PWD}:/work" -w /work node:22-alpine node verify.mjs k8s-rolloutThe last step adds --after to the end.
The verifier runs on your machine, so this page cannot see the result for itself: what you paste is what it records.
Checks
First run
- The cluster is running and its node is Readynot yet checked
- Every web pod is readynot yet checked
- The container has a readiness probenot yet checked
- The Service routes to the ready podsnot yet checked
- /data is a volume OpenTofu managesnot yet checked
- A check file was writtennot yet checked
The final run (--after)
- The cluster is running and its node is Readynot yet checked
- Every web pod is readynot yet checked
- The container has a readiness probenot yet checked
- The Service routes to the ready podsnot yet checked
- /data is a volume OpenTofu managesnot yet checked
- The first run was recordednot yet checked
- A bad rollout and a rollback happenednot yet checked
- The previous image is backnot yet checked
- The check file survivednot yet checked