Red monitor, healthy service

Symptom

Uptime Kuma showed one red monitor out of 66: the push heartbeat for my missed-call text-back service. Its history showed it had gone red within minutes of being created.

Impact

None to customers. The service was healthy and answering the whole time. The real risk was the monitoring gap: while the heartbeat was broken, a real outage would have looked exactly the same, and a monitor that’s always red teaches you to ignore red.

Triage

1. Is the service actually down? Ran its health check by hand: 200 ok. No.
2. Can the monitor receive? Sent a push to Kuma by hand: {"ok":true}, and the monitor flipped green. So Kuma was fine.
3. Is the job running? Cron logs showed it firing every minute, on schedule.
4. So the job ran, both halves worked by hand, and nothing arrived. That points at the environment the job runs in.

Root cause

The heartbeat is a cron one-liner on the hypervisor: run the health check inside the container with pct exec, and only if it passes push “up” to Kuma. pct lives in /usr/sbin. Cron runs with a minimal PATH that doesn’t include /usr/sbin, so the first command failed with “command not found”. The && chain stopped, and all output was sent to /dev/null. Every run failed without leaving a trace.

The original setup was “verified” by running the line by hand in an interactive root shell, which has a full PATH. The test environment didn’t match the real one.

Fix

Added an explicit PATH= line to the cron file. Confirmed with env -i PATH=/usr/bin:/bin that the old environment really couldn’t find pct.

Verification

The next two scheduled runs (no manual pushes) both landed as up / OK in Kuma. 66/66 monitors green.

Prevention

• Test cron jobs under cron’s environment (env -i), not your shell.
• Don’t discard all output in a heartbeat. Log failures somewhere, even if it’s just logger.
• A monitor that’s red from day one is a bug in the monitor until proven otherwise.