Alert when a worker stops making progress

Signals in productionAbout 40 minDraft

Not started.

What you need

Windows 11 with Docker Desktop running, and either PowerShell or a WSL terminal. The verifier runs inside a throwaway Node container, so nothing else needs installing.

Steps

  1. Save the three starting files and verify.mjs into an empty folder. Do not change worker.py.
  2. Start it with docker compose -p skilltree-alert up -d, open http://127.0.0.1:9090 and find worker_jobs_processed_total. Run the verifier: it fails, because there is no alert yet.
  3. Write rules.yml with an alert that fires when the worker has finished no jobs for a while, with a for: long enough that one slow job does not page anyone. Add it under rule_files: in prometheus.yml, mount it in compose.yaml, and restart Prometheus.
  4. Run the verifier while the worker is working: the alert must be loaded and quiet.
  5. Pause the worker with docker compose -p skilltree-alert exec -T worker touch /tmp/pause. Wait about two minutes.
  6. Run the verifier with --after at the end: the alert must be firing. Paste its last line below.

Why it matters

A stuck worker often still answers health checks and still serves metrics: it is the work that stopped. Alert on the thing you care about, jobs finished, rather than on the process being up.

increase(worker_jobs_processed_total[1m]) is how many jobs finished in the last minute. When it is zero, nothing finished. A counter only goes up, so a restart that resets it shows as a brief dip that increase handles for you.

The for: duration is how long the condition must hold before the alert fires. Too short and every slow job pages someone; too long and a real outage waits. Write down why you chose yours.

When you are done

  • docker compose -p skilltree-alert down
  • Delete the folder

Files

Show compose.yaml
# The alert-on-stall task: a worker and Prometheus. Do not change the worker; write rules.yml and wire it in.
services:
  worker:
    image: python:3.12-alpine
    command: ["python", "-u", "/app/worker.py"]
    volumes:
      - ./worker.py:/app/worker.py:ro
  prometheus:
    image: prom/prometheus:v3.5.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "127.0.0.1:9090:9090"
Show prometheus.yml
# Prometheus scrapes the worker every five seconds. Add a rule_files entry for your alert, and mount the file.
global:
  scrape_interval: 5s
  evaluation_interval: 5s
scrape_configs:
  - job_name: worker
    static_configs:
      - targets: ["worker:8000"]
Show worker.py
# A practice worker for the alert-on-stall task. It "processes" a job every second and serves Prometheus metrics on
# :8000/metrics. Create /tmp/pause inside its container and it stops processing but keeps serving metrics, which is
# what a stuck worker looks like from outside.
import http.server, os, threading, time

processed = 0
def work():
    global processed
    while True:
        time.sleep(1)
        if not os.path.exists('/tmp/pause'):
            processed += 1
threading.Thread(target=work, daemon=True).start()

class Metrics(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/metrics':
            self.send_response(404); self.end_headers(); return
        body = ('# HELP worker_jobs_processed_total Jobs finished since start.\n'
                '# TYPE worker_jobs_processed_total counter\n'
                f'worker_jobs_processed_total {processed}\n').encode()
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain; version=0.0.4')
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass

http.server.ThreadingHTTPServer(('0.0.0.0', 8000), Metrics).serve_forever()

It asks Docker for what it needs and changes nothing else; read it before you run it. Mounting the Docker socket gives it the same control over Docker that you have.

Run the verifier

In the folder with your files, in PowerShell or in WSL:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "${PWD}:/work" -w /work node:22-alpine node verify.mjs alert-on-stall

The last step adds --after to the end.

The verifier runs on your machine, so this page cannot see the result for itself: what you paste is what it records.

Checks

First run

  • The db service is runningnot yet checked
  • Prometheus scrapes the workernot yet checked
  • An alert on jobs finished is loadednot yet checked
  • It waits before firingnot yet checked
  • It stays quiet while the worker worksnot yet checked

The final run (--after)

  • The db service is runningnot yet checked
  • Prometheus scrapes the workernot yet checked
  • An alert on jobs finished is loadednot yet checked
  • It waits before firingnot yet checked
  • The worker is pausednot yet checked
  • It fires for the stalled workernot yet checked
Esc
↑ ↓ to move↵ to open