Alert when a worker stops making progress
Not started.
What you need
Windows 11 with Docker Desktop running, and either PowerShell or a WSL terminal. The verifier runs inside a throwaway Node container, so nothing else needs installing.
Steps
- Save the three starting files and
verify.mjsinto an empty folder. Do not changeworker.py. - Start it with
docker compose -p skilltree-alert up -d, open http://127.0.0.1:9090 and findworker_jobs_processed_total. Run the verifier: it fails, because there is no alert yet. - Write
rules.ymlwith an alert that fires when the worker has finished no jobs for a while, with afor:long enough that one slow job does not page anyone. Add it underrule_files:inprometheus.yml, mount it incompose.yaml, and restart Prometheus. - Run the verifier while the worker is working: the alert must be loaded and quiet.
- Pause the worker with
docker compose -p skilltree-alert exec -T worker touch /tmp/pause. Wait about two minutes. - Run the verifier with
--afterat the end: the alert must be firing. Paste its last line below.
Why it matters
A stuck worker often still answers health checks and still serves metrics: it is the work that stopped. Alert on the thing you care about, jobs finished, rather than on the process being up.
increase(worker_jobs_processed_total[1m]) is how many jobs finished in the last minute. When it is zero, nothing finished. A counter only goes up, so a restart that resets it shows as a brief dip that increase handles for you.
The for: duration is how long the condition must hold before the alert fires. Too short and every slow job pages someone; too long and a real outage waits. Write down why you chose yours.
When you are done
docker compose -p skilltree-alert downDelete the folder
Files
Show compose.yaml
# The alert-on-stall task: a worker and Prometheus. Do not change the worker; write rules.yml and wire it in.
services:
worker:
image: python:3.12-alpine
command: ["python", "-u", "/app/worker.py"]
volumes:
- ./worker.py:/app/worker.py:ro
prometheus:
image: prom/prometheus:v3.5.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
ports:
- "127.0.0.1:9090:9090"
Show prometheus.yml
# Prometheus scrapes the worker every five seconds. Add a rule_files entry for your alert, and mount the file.
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: worker
static_configs:
- targets: ["worker:8000"]
Show worker.py
# A practice worker for the alert-on-stall task. It "processes" a job every second and serves Prometheus metrics on
# :8000/metrics. Create /tmp/pause inside its container and it stops processing but keeps serving metrics, which is
# what a stuck worker looks like from outside.
import http.server, os, threading, time
processed = 0
def work():
global processed
while True:
time.sleep(1)
if not os.path.exists('/tmp/pause'):
processed += 1
threading.Thread(target=work, daemon=True).start()
class Metrics(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path != '/metrics':
self.send_response(404); self.end_headers(); return
body = ('# HELP worker_jobs_processed_total Jobs finished since start.\n'
'# TYPE worker_jobs_processed_total counter\n'
f'worker_jobs_processed_total {processed}\n').encode()
self.send_response(200)
self.send_header('Content-Type', 'text/plain; version=0.0.4')
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
http.server.ThreadingHTTPServer(('0.0.0.0', 8000), Metrics).serve_forever()
It asks Docker for what it needs and changes nothing else; read it before you run it. Mounting the Docker socket gives it the same control over Docker that you have.
Run the verifier
In the folder with your files, in PowerShell or in WSL:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "${PWD}:/work" -w /work node:22-alpine node verify.mjs alert-on-stallThe last step adds --after to the end.
The verifier runs on your machine, so this page cannot see the result for itself: what you paste is what it records.
Checks
First run
- The db service is runningnot yet checked
- Prometheus scrapes the workernot yet checked
- An alert on jobs finished is loadednot yet checked
- It waits before firingnot yet checked
- It stays quiet while the worker worksnot yet checked
The final run (--after)
- The db service is runningnot yet checked
- Prometheus scrapes the workernot yet checked
- An alert on jobs finished is loadednot yet checked
- It waits before firingnot yet checked
- The worker is pausednot yet checked
- It fires for the stalled workernot yet checked