Cron job monitoring

What this tool checks

A scheduled job, such as a nightly backup, can't be checked from the outside. When it silently stops running, nobody notices until the backup is needed.

A heartbeat monitor turns this round. You give the job a name, say how often it runs and how late it may be (the grace time), and you get a secret ping URL such as https://thenetworkery.com/hb/…. The job calls that URL each time it finishes. If no ping arrives within the expected period plus the grace time, the monitor goes down and we email you. When the next ping arrives, it goes back up and you get a recovery email.

The ping URL accepts GET, HEAD and POST, needs no headers or authentication, and answers 200 OK. Two extra addresses are available:

• /start at the end of the URL marks the beginning of a run. The next normal ping records how long the run took. A run that sent /start must finish within the grace time, or the monitor goes down.
• /fail reports that the job failed. The monitor goes down at once, and you are emailed.

With POST, the first 1,000 bytes of the body are stored with the ping, so a failing job can send the end of its log. Each monitor keeps its last 100 pings (time, method, run time, user agent and that body excerpt); older ones are deleted. We don't store the address the ping came from.

A monitor waits for its first ping before it starts timing, so you aren't alerted while setting the job up. Deadlines are checked every minute. Heartbeats count towards the 10 monitors of a free account.

Any of your monitors, including uptime, certificate and heartbeat monitors, can also be shown on a public status page at /status/your-name. The page shows each monitor's current state, a bar for each of the last 90 days and recent incidents. You choose whether it shows the targets (addresses, host names and check details) and whether search engines may list it.

Reading the results

The monitor's page shows its state:

Below that you see the expected period, the grace time, the time of the last ping and the time after which the job counts as late. The ping history lists every recent ping as Success, Failure or Started, with the run time when the job used /start, and any body it sent.

On a status page, each day's bar is green when the monitor was up all day, amber for a warning or an outage that kept the day above 99% uptime, red below 99%, and grey where there is no data. The uptime percentage counts warnings as up. Times are in UTC.

Common problems and fixes

The ping is sent even when the job failed

A cron line such as job.sh; curl URL pings whatever happened. Chain the commands with &&, so the ping only happens when the job exits with status 0:

30 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://thenetworkery.com/hb/your-token

Better still, call /fail when it fails, so you hear about it straight away rather than after the grace time.

Alerts arrive although the job ran

Check that the period matches the schedule and that the grace time covers how long the job takes and any delay in starting it. A job that runs daily at a random time needs a grace time of several hours. Also check that the server running the job can reach the internet: curl -fsS prints an error when the request fails.

The started run is reported as not finished

After /start, the job has the grace time to finish. Raise the grace time above the longest normal run time, or don't use /start for jobs whose run time varies widely.

Someone else could send pings

Anyone who knows the URL can ping it, which could hide a real failure. Keep the URL out of public repositories and logs. If it leaks, use "New URL" on the monitor's page: the old address stops working at once.

FAQ