Monitoring, the simple version

Something that texts you when a thing stops answering, and the one way it will lie to you.

After the four things on the Ship page are running, you want to know when one of them stops. The simple version is one container and twenty minutes.

Uptime Kuma

docker run -d --name kuma --restart unless-stopped -p 127.0.0.1:3001:3001 \
  -v ~/kuma:/app/data louislam/uptime-kuma

Add a monitor per service, HTTP for anything with a page, a TCP port for the rest. Add your Telegram bot as the notification, the same bot the agent uses if you like. Set it to notify after two consecutive failures, not one, or a single slow poll pages you.

Monitor the always-on agent by monitoring what it depends on: the tmux session’s existence is hard to check from outside, but the channel it talks through is not. A Kuma “push” monitor is the trick for that. The agent, or a tiny hook, hits a URL every few minutes to say “alive,” and Kuma alerts when the heartbeats stop. That catches the case where the process is up and the agent is wedged.

The one way it lies

A monitor running on the same mini as the things it monitors will, when the mini is under heavy load, report every service as down at once. It is not measuring the services. It is measuring its own starvation. The tell is in the numbers: a check with a five-second timeout that reports a fifty-second response time was not timing the service, it was waiting for a CPU. Jeff’s crew got four phone notifications from exactly this on the morning this page was written, and nothing was down.

Two defences. When many things fail in the same second, suspect the monitor first, especially if one of them lives on a different machine. And if you can, run the monitor somewhere else, a Raspberry Pi, a five-dollar VPS, your phone on the tailnet, anything that is not the box being watched.

How you know it worked

Stop a container. Your phone buzzes within two check intervals. Start it. Your phone buzzes again saying it is back.

Read more