> ## Content Index
> Fetch the complete content index at: https://blog-preview.sudowl.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# My Backups Were Silently Dead for 7 Weeks: How I Check Backup Freshness Now
- URL: https://blog-preview.sudowl.com/backups-silently-dead-check-backup-freshness/
- Published: 2026-10-18T10:00:00.000Z
- Updated: 2026-10-25T17:35:47.000Z
- Description: Scheduled backups stopped in late August and nothing told me. The cleanup, the offsite fix, and a script that alerts when the newest backup is too old.
- Author: Lukas
- Tags: Linux, Security

I did a security audit on an old Plesk box I'm slowly moving away from. Updates, SSH config, open ports, the usual list. The thing that actually scared me wasn't on the security list at all.

The disk was at 97 %. And the scheduled Plesk backups had stopped running in late August. That's about seven weeks without a single new backup. The box hosts a community forum I run and the website and mail of a small business I look after. Neither had a current backup, and nothing was offsite.

Nobody got a mail. Nothing turned red. The backups just stopped, and everything else kept working, so there was no reason to look.

**TL;DR:** Don't monitor that the backup job ran, monitor that a fresh backup exists. Check the age of the newest file, compare its size to the previous one, test the archive, and send failures to a place you actually read (Discord webhook, chat, mail if the box can send mail). Run the check on the backup target, not only on the source. And do a real test restore now and then, because a file that exists is not a backup you can restore.

## What happened

The Plesk license on that box had expired. Sites and mail keep running without a license, I wrote about that in [what still works when a Plesk license expires](/plesk-expired-license-what-still-works/). The scheduled full backups don't. They quietly stopped on August 26 and that was it.

At the same time the disk filled up. The breakdown:

- about 25 GB of old Plesk backup dumps
- about 14 GB of Docker data, mostly stopped containers of apps I had already moved to another server
- about 7.7 GB of leftover app directories from the same move

That left 2.8 GB free on a 75 GB disk. If you need to find out where your space went, I have a whole post on [finding what's eating your disk](/linux-disk-full-forensics/).

The audit also showed the box was still on the kernel it had booted in July. Newer kernels were installed but never active, because automatic reboots were commented out in unattended-upgrades. Patches installed, none running.

## The fix, in order

**1\. Free the disk.** I wrote a small cleanup script with a `plan` mode that only prints what it would delete, and an `apply` mode that does it. Then I ran `plan`, read it, ran `apply`. In essence it does this:

```bash
docker container prune -f
docker image prune -a -f
rm /var/lib/psa/dumps/<two old full dumps from the summer>

```

Disk went from 97 % to 64 %, 27 GB free.

**2\. Database dumps that don't depend on the panel.** A plain cron job that dumps the forum database every night and keeps 14 days. Roughly this, with the MySQL credentials in `/root/.my.cnf`:

```bash
# /etc/cron.d/forum-db-backup
41 3 * * * root mysqldump --single-transaction forumdb | gzip > /backup/db/daily-$(date -u +\%F).sql.gz && find /backup/db -name 'daily-*.sql.gz' -mtime +14 -delete

```

Note the `\%` in cron. Without the backslash cron treats `%` as a newline and the job breaks in a confusing way. I tested the first dump by hand before I trusted it.

**3\. Reboot.** Into the current kernel. Then I could stop worrying about that one.

**4\. Offsite.** This is the part that was really missing. A script on the box runs every night at 04:15\. It dumps all databases, including the one inside a Docker container, and rsyncs the vhosts, the mail directories, `/etc` and the app directories to another server. The target user is locked down with `rrsync` and its own SSH key, so the source can only write into its own backup folder and can't run anything else there.

On the target, a second job at 06:30 makes a hardlink snapshot of the current copy and keeps 14 of them:

```bash
cp -al /backup/source/current "/backup/source/snap/$(date -u +%F)"
find /backup/source/snap -mindepth 1 -maxdepth 1 -type d -mtime +14 -exec rm -rf {} +

```

`cp -al` copies the directory tree but hardlinks the files, so a snapshot of unchanged files costs almost no space. The first full run was about 9 GB. The basics of this setup are in [rsync backups on Linux](/rsync-backup-linux/).

## Gotcha: the snapshot side already knew, and told nobody

The snapshot job on the target checks a `LAST_OK` stamp that the source writes after a successful run, and marks the backup as STALE if it's older than 20 hours. Good idea. But it writes that into a log file. And a log nobody reads is exactly how I got seven dead weeks in the first place.

So the real lesson isn't "have backups". It's "have something that complains when the backups stop, in a place you look every day".

## The freshness check

This is the check that goes on top of it. It looks at a backup directory, takes the newest file, and complains if it's too old, empty, a lot smaller than the one before, or fails an integrity test. Failures go to a Discord webhook.

```bash
#!/usr/bin/env bash
# backup-freshness.sh: complain if the newest backup is old, small or broken
set -euo pipefail

BACKUP_DIR="${BACKUP_DIR:?set BACKUP_DIR}"
PATTERN="${PATTERN:-*}"
MAX_AGE_H="${MAX_AGE_H:-26}"      # daily backup + some slack
MIN_PCT="${MIN_PCT:-80}"          # newest must be >= 80 % of the previous one
WEBHOOK="${WEBHOOK:?set WEBHOOK}"
WHERE="$(hostname -s):$BACKUP_DIR"

alert() {
  local msg="Backup check failed on $WHERE: $1"
  echo "$msg" >&2
  curl -fsS -m 15 -H 'Content-Type: application/json' \
    -d "$(jq -n --arg c "$msg" '{content: $c}')" \
    "$WEBHOOK" >/dev/null || echo "webhook post failed too" >&2
  exit 1
}

[ -d "$BACKUP_DIR" ] || alert "directory does not exist"

# newest two files: "mtime size path"
mapfile -t newest < <(
  find "$BACKUP_DIR" -maxdepth 1 -type f -name "$PATTERN" -printf '%T@ %s %p\n' \
    | sort -rn | head -n 2
)
[ "${#newest[@]}" -gt 0 ] || alert "no files matching '$PATTERN'"

read -r ts size path <<<"${newest[0]}"
age_h=$(( ( $(date +%s) - ${ts%.*} ) / 3600 ))

[ "$age_h" -le "$MAX_AGE_H" ] || alert "newest file is ${age_h}h old: $path"
[ "$size" -gt 0 ] || alert "newest file is empty: $path"

if [ "${#newest[@]}" -eq 2 ]; then
  read -r _ prev_size _ <<<"${newest[1]}"
  if [ "$prev_size" -gt 0 ]; then
    pct=$(( size * 100 / prev_size ))
    [ "$pct" -ge "$MIN_PCT" ] || alert "newest is ${pct} % of the previous one ($size vs $prev_size bytes): $path"
  fi
fi

case "$path" in
  *.zst) zstd -tq "$path" || alert "zstd test failed: $path" ;;
  *.gz)  gzip -t "$path"  || alert "gzip test failed: $path" ;;
esac

echo "ok: $path, ${age_h}h old, $size bytes"

```

It needs `curl` and `jq` (`apt install jq`). `jq` builds the JSON so a weird file name can't break the payload. The webhook URL goes into an env file with `chmod 600`, because anyone with that URL can post into your channel.

Why each check:

- **Age** catches the job that stopped running. That's my seven weeks.
- **Size vs. previous** catches the job that still runs but backs up less. An empty database dump, a mount that wasn't there, a tar that skipped a directory because of permissions.
- **Integrity test** catches the truncated upload. I lost data to exactly that once, the story is in [my backup post-mortem](/server-provider-outage-backup-lessons/).

For a directory backup like the rsync copy above there's no single newest file. Use a stamp instead. The backup script writes `date +%s > LAST_OK` as its very last step, only if everything before it worked, and the check reads it:

```bash
last=$(cat "$BACKUP_DIR/LAST_OK" 2>/dev/null || echo 0)
age_h=$(( ( $(date +%s) - last ) / 3600 ))
[ "$age_h" -le "$MAX_AGE_H" ] || alert "LAST_OK is ${age_h}h old"

```

## Gotcha: rsync keeps the old mtime

`rsync -a` preserves modification times. If you copy a backup file to the target with `rsync -a`, its mtime is when it was created on the source, which is fine. But if you check the age of a whole synced tree, unchanged files keep their old dates forever, and some of them will look "stale" even when the sync ran an hour ago. For trees, check the stamp, not the files.

## Run it from a timer, on the target

Run it as a systemd timer, after the snapshot job is done:

```ini
# /etc/systemd/system/backup-freshness.service
[Unit]
Description=Check that the newest backup is fresh

[Service]
Type=oneshot
EnvironmentFile=/etc/backup-freshness.env
ExecStart=/usr/local/bin/backup-freshness.sh

```

```ini
# /etc/systemd/system/backup-freshness.timer
[Unit]
Description=Daily backup freshness check

[Timer]
OnCalendar=*-*-* 07:30:00 UTC
Persistent=true

[Install]
WantedBy=timers.target

```

```bash
systemctl daemon-reload
systemctl enable --now backup-freshness.timer
systemctl start backup-freshness.service   # run once now and look at the output

```

Why timers and not cron, see [systemd timers instead of cron](/systemd-timers-instead-of-cron/). For several backup sets, make it a template unit or just call the script a few times with different env files.

Put the check on the backup target or a third box. If the source server is the one that dies, it can't tell you its backups stopped.

## Gotcha: the alert path needs to work too

Mail is the classic choice, but a lot of small boxes have no working MTA, and some providers block outbound port 25\. Then `mail -s` "works" and the mail goes nowhere. That's why I use a Discord webhook, same idea as my [Discord alerts from server logs](/discord-alerts-server-logs-systemd/). Whatever you use, trigger a fake failure once (set `MAX_AGE_H=0`) and check that the message actually arrives.

And if the checker itself stops running, you're back to silence. A dead man's switch fixes that: the script pings a URL on success, and the other side alerts when the ping doesn't come. Healthchecks does this and you can self-host it.

## Test restores

All of the above only proves a file is there and looks sane. Once a month I'd pull the newest database dump into a throwaway container and see if it loads:

```bash
docker run -d --name restore-test -e MARIADB_ROOT_PASSWORD=throwaway mariadb:10.11
until docker exec restore-test mariadb-admin ping -uroot -pthrowaway --silent; do sleep 2; done

docker exec restore-test mariadb -uroot -pthrowaway -e 'CREATE DATABASE t'
zcat /backup/db/daily-2026-10-18.sql.gz | docker exec -i restore-test mariadb -uroot -pthrowaway t
docker exec restore-test mariadb -uroot -pthrowaway t -e 'SHOW TABLES' | wc -l

docker rm -f restore-test

```

If the dump was made with `--databases`, it contains its own `CREATE DATABASE` and `USE`, and the import ignores the `t`. Then look in the database it created instead.

For files: extract one site from the snapshot into `/tmp` and open a couple of files. Ten minutes a month. Way cheaper than finding out the hard way.

## Disk is part of it

A full disk kills backups too, and the backups are often what fills it. Two more lines in the same script, before the final `echo`, and you get warned before that happens:

```bash
use=$(df --output=pcent / | tail -1 | tr -dc '0-9')
[ "$use" -lt 90 ] || alert "disk / at ${use} %"

```

97 % was my wake-up call. I'd rather get a Discord message at 90.

---

### Related posts

- [Mail + Web on One Server: 4 GB or 8 GB of RAM? I Measured It](https://blog.sudowl.com/mail-server-ram-budget-4gb-vs-8gb/?ref=blog-preview.sudowl.com)
- [Postfix SNI: "map lookup problem" and TLS Alert 80 — the texthash Fix](https://blog.sudowl.com/postfix-sni-hash-map-lookup-problem/?ref=blog-preview.sudowl.com)
- [Plesk License Expired: What Still Works and How I Got Around It](https://blog.sudowl.com/plesk-expired-license-what-still-works/?ref=blog-preview.sudowl.com)