Heartbeat Monitoring for Cron Jobs (Catching the Job That Quietly Stopped)

Most monitoring watches things that are supposed to answer. A cron job is the opposite: it's supposed to do its work and stay quiet. So when it stops running, there's nothing to see. No error in the log, because the job never started. No failed request, because nothing made one. The nightly backup just doesn't happen, and you find out three weeks later, the night you actually need to restore from it.

Every scheduled task fails this way. Backups, billing reconciliation, the data export your warehouse depends on, the 6 a.m. digest email: they fail by absence. A job that throws an exception at least leaves a stack trace behind. A job that silently stops getting scheduled leaves nothing at all, and a monitor pointed at your live service has no way to see a thing that didn't run.

Watch for the ping that doesn't arrive

The fix turns the check around. Rather than your monitor reaching out to probe a service, the job reaches in to say it ran. You get a unique URL, the job hits it at the end of every successful run, and the monitor's whole purpose is to notice when an expected ping fails to show. The pattern has an old name: a dead man's switch, where silence is the alarm.

The logic underneath is plain. You tell the monitor how often the job should check in. A ping that lands inside that window means all's well and nothing happens; a window that passes with no ping means the monitor goes down and starts alerting, because the only reason a healthy job goes quiet is that it stopped.

Grace periods and the cold-start problem

Two details separate a heartbeat monitor that earns its keep from one that cries wolf.

The first is the grace period. A backup that usually wraps at 2:00 sometimes wraps at 2:09 because the disk was busy, and a page for nine minutes of ordinary variance trains you to ignore the thing. The grace period is slack stacked on top of the interval: a daily job with a 30-minute grace won't alert until it's a full half-hour late, covering normal jitter without papering over a run that genuinely failed.

The second is what happens the moment you create the monitor. A fresh heartbeat hasn't been pinged yet, and a naive monitor would call that "down" on the spot and page you for a job whose first scheduled run hasn't come around. failover.io holds a new heartbeat in a pending state until its first ping lands; only then does the clock start ticking. You set it up, deploy the job, and the monitor arms itself when the first real run checks in. Nothing false at creation time.

Setting it up

Create a heartbeat monitor, set the interval to match the schedule (hourly, daily at 2 a.m., whatever your crontab says), and set a grace period that covers the job's normal slack. Copy the ping URL it hands you and tack it onto the end of the job:

The placement is the entire trick. Put the curl after your real command with &&, so the ping fires only when the job actually succeeded. Now a single check catches both failure modes: the job not running at all, where no ping arrives and the window expires, and the job running but failing, where the command errors, the && short-circuits, and the ping never goes out. Ping unconditionally and you'd only ever catch the first.

A missed heartbeat is an ordinary "down" to the rest of the system. It opens an incident and climbs the escalation chain like any outage, from Slack to email to SMS to a phone call, until someone acknowledges. When the job runs again and pings, that first ping after a down sends a recovery alert, so you learn it's back without opening the dashboard.

The short version

Scheduled jobs fail silently because their failure mode is doing nothing, and nothing is exactly what a service-facing monitor can't see. Have the job report in on every successful run and alert on the ping that never comes. Set a grace period for normal lateness, let the monitor arm on its first real ping so creating it doesn't page you, and put the curl after the command so a failed run reads the same as a job that never started. The 2 a.m. backup that quietly stopped turns into a phone call that night, instead of a discovery three weeks on.