Get alerted when your Supabase backup to S3 fails, or never runs
Exit codes, a dead man's switch, and a daily Lambda that checks S3 for missing or overdue backups.
This follows an earlier post, Back up a Supabase database to Amazon S3 on a schedule, and prove it restores , which set up a bucket, an IAM key that can write backups but not delete them, a weekly backup in GitHub Actions and a restore drill. This post is about the next question: how do you find out when that job stops working?
A scheduled backup can go wrong in three ways:
- It fails, and says so.
pg_dumpcan't connect, the key was rotated, the drill finds an empty table. The job exits non-zero.
- It fails where nobody looks. Same failure, but the exit code lands in a log or a notification nobody reads.
- It doesn't run. Nothing fails, so nothing reports. The schedule was switched off, the machine was asleep, or someone renamed the project and the backups now go to a different folder.
The first is easy. The third is the one you find out about on the day you need a restore. This post adds one check for each, using the open-source
backupdrill CLI from that post (MIT licensed, on npm) and a few AWS services. You don't need all three. Pick the ones that fit how you run backups.Layer 1: let the job fail loudly
Both commands report failure through their exit code.
backupdrill backup exits non-zero when the backup fails. backupdrill drill exits with code 1 when any check fails, such as a checksum that doesn't match or a table that came back empty. If you ask for Auth users with --auth and they can't be included, the database backup still completes and the command exits with code 3, so a partial backup doesn't pass as a full one.In GitHub Actions a non-zero exit fails the job, and GitHub can email someone about it. For a scheduled workflow, that's the user who last changed the
cron line in the workflow file, and only if their Actions email notifications are turned on. If that person has left, has notifications off or filters GitHub mail, the alert goes nowhere. Check who last edited the schedule and what their notification settings are, or rely on the next two layers.That's all this layer gives you. It can't tell you about a run that never started.
Layer 2: a dead man's switch with Healthchecks.io
A dead man's switch works the other way round. Instead of waiting for an error, it expects to hear from your job on a schedule, and silence is enough to raise an alert. Healthchecks.io is a common choice: the free Hobbyist plan covers 20 checks, and it's open source, so you can also run your own.
The CLI has this built in (version 2.5.0 and later). Give it a ping URL and every
backup or drill run sends:-
<url>/startwhen the run begins
-
<url>when it succeeds, with a one-line summary: table count, Storage file count and the snapshot name
-
<url>/failwhen it fails, with the error in the body
Both pings carry the same run ID, so overlapping runs don't get mixed up.
Set it up
- In Healthchecks.io, add a check. Give it the same schedule as the job. For the weekly workflow from the earlier post, use a Cron schedule of
0 3 * * 0in UTC.
- Set Grace Time generously. Because the CLI sends a start ping, Healthchecks also treats Grace Time as the longest a run may take. Scheduled GitHub Actions runs can start late, so add a margin to your slowest run.
- Copy the ping URL (
https://hc-ping.com/<uuid>) and add it as a repository secret namedBACKUPDRILL_PING_URL. Treat it as a secret: anyone who has it can tell Healthchecks your backup ran.
- Pass it to the action:
- uses: backupdrill/supabase-backup-action@v1 with: database-url: ${{ secrets.BACKUPDRILL_DATABASE_URL }} s3-bucket: ${{ secrets.BACKUPDRILL_S3_BUCKET }} s3-access-key-id: ${{ secrets.BACKUPDRILL_S3_ACCESS_KEY_ID }} s3-secret-access-key: ${{ secrets.BACKUPDRILL_S3_SECRET_ACCESS_KEY }} s3-region: ${{ secrets.BACKUPDRILL_S3_REGION }} project-name: my-app drill: "true" ping-url: ${{ secrets.BACKUPDRILL_PING_URL }}The action sends the backup's pings and then the drill's to the same check. A drill that runs but doesn't pass sends
/fail with the failed checks in the body, so the alert says what broke.If you run the CLI from cron on an EC2 instance instead, set
BACKUPDRILL_PING_URL in the environment file the job already loads, or pass --ping-url . If backups and drills run on separate schedules there, give each its own check. On a shared check, the backup's success pings keep it green while a drill that stopped running goes unnoticed.What gets sent, and what happens if Healthchecks is down
A failure message can include connection details, so the CLI filters it before sending. It replaces connection strings, passwords, hostnames, IP addresses, Supabase project refs and bucket keys with placeholders, and drops the parts of Postgres errors that usually quote row values or SQL (
DETAIL , CONTEXT , Command was ). It's a filter, not a guarantee, so only send these pings somewhere you'd also be fine pasting the job log.A ping never changes the result of a run. Each request times out after 10 seconds and is retried twice on network errors and 5xx responses. If it still fails, the CLI prints a warning and exits with the same code it would have had anyway. A lost success ping means a false alarm for a run that worked. A lost failure ping delays the alert until the check's schedule and grace time run out.
To test it, run the workflow once by hand from the Actions tab. The check should show a start event, then a success with the run time. On the Integrations page in Healthchecks, the Test! button sends a sample alert, which is worth doing once to make sure it reaches a channel you actually read.
Layer 3: check the bucket from inside AWS
The first two layers watch the job. This one looks at the result: is there a recent, complete snapshot in the bucket? It catches the cases the job can't report on, such as a renamed project, a secret pointing at a different bucket, or a workflow someone deleted, and it runs in your AWS account, so it doesn't depend on GitHub or a third-party service being up.
It relies on one detail of how the CLI writes a snapshot. Each backup creates a folder named after the UTC time it started, such as
backupdrill/my-app/2026-11-07T11-35-38-795Z/ , and writes manifest.json into it last. A folder without a manifest is a backup that didn't finish. The folder names sort in time order, so the newest complete snapshot is the last folder that has a manifest.The pieces:
- an SNS topic with an email subscription, for the alert
- a Lambda function that finds the newest complete snapshot and publishes to the topic if it's too old
- an EventBridge Scheduler schedule that runs the function once a day
Step 1: the SNS topic
In the SNS console, create a Standard topic named
supabase-backup-alerts , then create a subscription with protocol Email and your address. SNS sends a confirmation email, and nothing is delivered until you click the link in it. Copy the topic ARN.Step 2: the function
Create a Lambda function with a Python 3.13 runtime and paste this in. It uses only boto3, which the runtime includes.
import osfrom datetime import datetime, timezone
import boto3from botocore.exceptions import ClientError
s3 = boto3.client("s3")sns = boto3.client("sns")
BUCKET = os.environ["BUCKET"]# The CLI's default prefix plus your project name, with a trailing slash.SNAPSHOT_PREFIX = os.environ.get("SNAPSHOT_PREFIX", "backupdrill/my-app/")MAX_AGE_HOURS = float(os.environ.get("MAX_AGE_HOURS", "192"))TOPIC_ARN = os.environ["TOPIC_ARN"]
def snapshot_folders(): paginator = s3.get_paginator("list_objects_v2") for page in paginator.paginate(Bucket=BUCKET, Prefix=SNAPSHOT_PREFIX, Delimiter="/"): for common_prefix in page.get("CommonPrefixes", []): yield common_prefix["Prefix"]
def newest_complete_snapshot(): # Folder names are UTC timestamps, so sorting them sorts by time. for folder in sorted(snapshot_folders(), reverse=True): try: manifest = s3.head_object(Bucket=BUCKET, Key=folder + "manifest.json") except ClientError as error: if error.response["Error"]["Code"] in ("404", "NoSuchKey", "NotFound"): continue # this backup never finished; look at the one before it raise return folder, manifest["LastModified"] return None, None
def lambda_handler(event, context): folder, finished_at = newest_complete_snapshot() location = f"s3://{BUCKET}/{SNAPSHOT_PREFIX}" if folder is None: problem = f"No complete snapshot found under {location}" else: age_hours = (datetime.now(timezone.utc) - finished_at).total_seconds() / 3600 if age_hours <= MAX_AGE_HOURS: return {"status": "ok", "snapshot": folder, "age_hours": round(age_hours, 1)} problem = ( f"The newest complete snapshot under {location} is {age_hours:.0f} hours old " f"(limit {MAX_AGE_HOURS:.0f}): {folder}" ) sns.publish(TopicArn=TOPIC_ARN, Subject="Supabase backup is overdue", Message=problem) return {"status": "alert", "problem": problem}The function name matches the console's default handler,
lambda_function.lambda_handler . Add three environment variables: BUCKET , SNAPSHOT_PREFIX (for the earlier post's setup, backupdrill/my-app/ ) and TOPIC_ARN . MAX_AGE_HOURS defaults to 192, which is a weekly schedule plus a day of slack. For daily backups, set it to 26 or so.The function lists folder names only (the
Delimiter="/" part), not every object, so it stays cheap even when snapshots include thousands of Storage files. The default 3-second timeout can be short for a bucket with years of snapshots, so raise it to 30 seconds.Step 3: permissions
The function's execution role needs read access to the snapshot folders and permission to publish to one topic. Add this inline policy next to the basic execution role Lambda created for logging, with your bucket, prefix and topic ARN:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-backups" }, { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-backups/backupdrill/my-app/*" }, { "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:us-east-1:123456789012:supabase-backup-alerts" } ]}HeadObject is covered by s3:GetObject . s3:ListBucket matters for a second reason: without it, S3 answers a request for a missing manifest.json with 403 instead of 404, and the function would fail on an unfinished backup instead of skipping it.This role can read your backups, so keep it separate from the backup key in the earlier post. That key can write and can't delete, and this one can read and can't write.
Step 4: run it every day
In the EventBridge console, open Scheduler → Schedules → Create schedule. Choose a recurring, cron-based schedule, for example
0 9 * * ? * (09:00 every day; EventBridge cron has six fields), set the time zone you want, pick AWS Lambda Invoke as the target and select the function. Let the console create the schedule's execution role.Daily works even for weekly backups. The threshold decides when to alert, and a daily check means you hear about it within a day of the backup becoming overdue.
Step 5: an alarm for the function's own errors
If the function fails, for instance because someone removed its permissions, it raises an error instead of publishing, and you'd hear nothing. Add a CloudWatch alarm on the function's
Errors metric (namespace AWS/Lambda ), threshold greater than 0 over one day, with the same SNS topic as the alarm action. This covers a function that runs and fails. It doesn't cover a schedule that someone disabled, so leave the schedule alone or check it when you review the setup.Test it
Run the function once from the console with an empty test event. With a fresh backup in the bucket it returns
"status": "ok" and the snapshot name. Then set MAX_AGE_HOURS to 0 , run it again and check that the email arrives. Set it back afterwards.If you already run Prometheus
If you have Prometheus somewhere, the CLI can replace the Lambda.
backupdrill exporter serves metrics read from the bucket, including backupdrill_backup_last_success_timestamp_seconds for the newest complete snapshot. Run drills with --record and it also reports when the last drill ran and whether it passed. The CLI's README has alert rules to start from, and there's a longer walk-through in the Prometheus guide . The exporter only reads the bucket, so a key with read-only access to it is enough.What you have now
- a job that fails loudly, and GitHub's email about it if notifications are on for whoever last edited the schedule
- a Healthchecks check that alerts on a failed run and on a run that never happened
- a daily Lambda that checks the bucket itself and emails you through SNS when the newest complete snapshot is too old
- a CloudWatch alarm that emails you when that Lambda records an error
The Healthchecks guide goes further into grace times, what the alert email looks like and self-hosted Healthchecks.
Coming next
A follow-up post: restoring a snapshot into a new Supabase project, Auth users included, and what you still have to set up by hand afterwards.
I work on BackupDrill, the team behind the CLI. Everything in this post uses the open-source CLI and your own AWS account.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article