Backups & Restore

AnvilBase supports on-demand and scheduled logical backups per project (via the API/CLI/console), per-project retention pruning, automated restore verification, and standard pg_dump-based backups for the whole deployment. For continuous, sub-minute-RPO protection, layer Point-in-Time Recovery on top.

Per-project backups (API / CLI / console)

A project backup runs pg_dump and uploads the result to storage, asynchronously.

# Create (returns immediately with a pending record; runs in the background)
anvilbase backup create --project <id>

# List (most recent first)
anvilbase backup list --project <id>

# Restore from a completed backup
anvilbase backup restore <backup_id> --project <id>

API equivalents:

curl -X POST http://localhost:39001/api/v1/projects/<id>/backups -H "Authorization: Bearer $ANVILBASE_TOKEN"
curl http://localhost:39001/api/v1/projects/<id>/backups -H "Authorization: Bearer $ANVILBASE_TOKEN"
curl -X POST http://localhost:39001/api/v1/projects/<id>/backups/<backup_id>/restore -H "Authorization: Bearer $ANVILBASE_TOKEN"

In the console: project → Backups to trigger and restore.

Vault secrets survive restore — no pgsodium root key to preserve

A project’s logical backup excludes the pgsodium schema (its key material must never travel in a portable dump). This drops nothing your vault needs: AnvilBase does not rely on the pgsodium root key (/etc/postgresql-custom/pgsodium_root.key, random per boot, outside the postgres_data volume) to protect secrets.

The per-project vault.secrets mirror (e.g. the project JWT secret) is encrypted at the application layer with a key derived from your deployment CONTROL_PLANE_SECRET (AES-256-GCM via HKDF), and stored as ordinary table data. A restore onto the same deployment decrypts it from CONTROL_PLANE_SECRET alone.
The control-plane developer secrets vault (project_secrets) lives in the platform DB and is encrypted with pgp_sym_encrypt keyed by the same CONTROL_PLANE_SECRET.

Operator requirement — CONTROL_PLANE_SECRET MUST remain stable: it is the only key that can decrypt your secrets. Store it in your secret manager and set it identically on every restore/upgrade target. There is no automated re-key path. Rotating CONTROL_PLANE_SECRET would strand every existing secret (all enc:v1: ciphertext — vault JWT secrets, SMTP/OAuth/webhook secrets, and the pgp_sym_encrypt-keyed project_secrets) because they were encrypted under the old key and nothing decrypts them with the old key and re-encrypts with the new one. A rotation would require a separate, deliberate decrypt-old / re-encrypt-new migration that does not exist today — do not attempt it via the convergence sweep below. You do not need to back up or restore the pgsodium root key.

Upgrading from a pre-fix deployment: older project DBs stored the JWT secret as plaintext in vault.secrets. New backups encrypt it; old plaintext is read transparently (no stranding). To converge existing project DBs to encrypted at-rest storage in one pass, run:

curl -X POST http://localhost:39001/api/v1/admin/encrypt-settings-secrets \
  -H "Authorization: Bearer $ANVILBASE_TOKEN"

The response includes vault_jwt_secrets_encrypted (count of project DBs migrated). This sweep is a one-time, idempotent convergence of legacy plaintext → enc:v1: only — it skips values already in enc:v1: form and therefore can never re-key existing ciphertext. It is not a CONTROL_PLANE_SECRET rotation tool.

Where backups live: encryption & key layout

Per-project dumps are uploaded to the shared anvilbase-backups bucket on the stack’s object store (MinIO). The control plane enforces SSE-S3 (AES256) default encryption on this bucket: the rule is applied when the bucket is created and re-applied (idempotently) before every upload, so deployments that predate this hardening are encrypted automatically on their next backup — no manual migration. If the rule cannot be applied (e.g. MinIO without MINIO_KMS_SECRET_KEY — the same requirement project data buckets already have, see Configuration), the upload fails rather than writing a plaintext dump.

Object keys are namespaced per project:

Layout	Key format	Written by
Current	`projects/<project_id>/<YYYYMMDD_HHMMSS>.dump`	all new backups
Legacy	`<project_id>/<YYYYMMDD_HHMMSS>.dump`	backups taken before the at-rest hardening

Legacy keys remain fully listable and restorable — nothing to migrate. Restore accepts both layouts but strictly rejects any key outside the requested project’s namespace (path traversal, absolute keys, another project’s prefix), so a restore request for project X can never read another project’s dump.

Offsite copy (optional)

Losing the MinIO volume must not mean losing every backup. Configure a secondary S3-compatible target and every successful backup is asynchronously copied there after the primary upload:

Variable	Required	Purpose
`BACKUP_OFFSITE_ENDPOINT`	yes*	S3 endpoint URL (AWS S3, second MinIO, B2, …)
`BACKUP_OFFSITE_BUCKET`	yes*	target bucket
`BACKUP_OFFSITE_ACCESS_KEY`	yes*	credentials for the target
`BACKUP_OFFSITE_SECRET_KEY`	yes*	credentials for the target
`BACKUP_OFFSITE_REGION`	no	SigV4 region, default `us-east-1` (set the real region for AWS S3)

* All four must be set together — a partial configuration is rejected at boot so a typo can’t silently disable your DR copy. With none set there is no behavior change at all.

Failure semantics:

The copy runs fire-and-forget after the backup record is already completed — an offsite outage can never block, slow down, or fail a backup.
Outcome is recorded in the audit log: backup.offsite_copied on success, backup.offsite_failed (with the error) on failure — alert on the latter.
Objects are copied under the same projects/<project_id>/… key. The control plane best-effort applies the AES256 SSE rule to the offsite bucket too, but won’t abort the copy if the target refuses it (AWS S3 encrypts new buckets by default).

Scheduled backups, retention & restore verification

Beyond on-demand backups, each project can run scheduled backups on a cadence, prune old backups automatically (retention), and have its newest dump automatically restore-verified into a throwaway database every night. All three are driven by the control plane directly — there is no extra cron container to operate.

How it works (no pg_cron)

The schedule lives in the platform database (backup_settings), and the control plane runs three things on a timer:

A scheduler that, for each enabled and due project, enqueues a pending backup row and advances its next_run_at (atomically, so multiple control-plane replicas never double-enqueue).
A backup worker that drains those enqueued rows (FOR UPDATE SKIP LOCKED) and runs the same pg_dump → bucket path as the on-demand API.
A retention pruner and a restore verifier (below).

Why not pg_cron? pg_cron runs SQL inside a project database, but a backup (pg_dump → object store) is a control-plane operation, and the schedule lives in the platform DB. A control-plane scheduler reading backup_settings directly needs no cross-database bridge and no cron-expression parser. pg_cron remains available for your own in-DB scheduled jobs; it’s just not how platform backups are scheduled.

Configure a schedule

Admin-scoped endpoints read and set the per-project policy:

# Read the current settings (defaults: disabled, daily, no retention)
curl http://localhost:39001/api/v1/projects/<id>/backups/settings \
  -H "Authorization: Bearer $ANVILBASE_TOKEN"

# Enable: daily backups, keep 14 days OR the newest 30 (whichever is stricter),
# and verify the newest dump nightly.
curl -X PUT http://localhost:39001/api/v1/projects/<id>/backups/settings \
  -H "Authorization: Bearer $ANVILBASE_TOKEN" -H "Content-Type: application/json" \
  -d '{
        "enabled": true,
        "interval_minutes": 1440,
        "retain_days": 14,
        "retain_count": 30,
        "verify_enabled": true,
        "verify_interval_minutes": 1440
      }'

Field	Meaning	Bounds
`enabled`	Run scheduled backups for this project	—
`interval_minutes`	Backup cadence (`1440` = daily, `60` = hourly, `10080` = weekly)	5 … 525600
`retain_days`	Prune completed backups older than this many days (`null` = no age pruning)	1 … 3650
`retain_count`	Keep only the newest N completed backups (`null` = no count pruning)	1 … 10000
`verify_enabled`	Run the nightly restore verification	—
`verify_interval_minutes`	Verification cadence	5 … 525600

Setting a schedule provisions nothing external and is idempotent — re-PUT to change it. A project without object storage configured won’t crash provisioning; its scheduled backups simply fail with a clear error (visible in the backup list) until storage is configured.

Retention pruning

When a project has a retain_days and/or retain_count policy, the pruner removes backups outside the window. A backup is pruned if it violates either bound (too old or beyond the newest N) — the stricter, safer reading. Pruning is careful by construction:

It only ever considers that project’s own completed backups (scoped by project_id, and every candidate’s object key is re-validated against the project’s namespace before any delete).
It deletes the object first, then the row, so a crash mid-prune can only leave a row whose object is already gone (harmless) — it never orphans an S3 object.
Safety floor: the single newest completed backup is never pruned, no matter how old. Age-only retention (retain_days set, retain_count unset) could otherwise prune a project’s last backup once it ages out, leaving the project with zero backups — so the most-recent backup is always kept.

With no retention set, nothing is ever pruned (it never prunes “everything”).

Automated restore verification

A backup you’ve never restored is a hope, not a backup. With verify_enabled, the verifier periodically:

Picks the project’s newest completed backup.
Restores it into a fresh scratch database (verify_<id>_<ts>) — never the live project DB. The scratch name is guarded so it can never be a live platform_* or maintenance database.
Runs a sanity query (the restored schema is loadable and queryable).
Always drops the scratch database, even on failure.
Records the outcome (passed / failed / skipped) in backup_verifications and the audit log (backup.verify.*).

This is strictly non-destructive — it never touches the live instance.

A failed verification is never silent: /health/services surfaces the top-level warning string backup_restore_verification_failed whenever any project’s most-recent verification failed. Alert on it the same way you alert on rate_limit_degraded_fallback_to_in_memory:

curl -s http://localhost:39001/health/services | jq '.warnings'
# [ "backup_restore_verification_failed" ]   ← page on this

Whole-deployment logical backups

Back up the platform database and every project database:

# Platform database
docker compose exec -T postgres pg_dump -U postgres -Fc anvilbase_platform > backup-platform.dump

# Each project database (platform_*)
docker compose exec postgres psql -U postgres -t -A \
  -c "SELECT datname FROM pg_database WHERE datname LIKE 'platform_%'" | \
while read db; do
  docker compose exec -T postgres pg_dump -U postgres -Fc "$db" > "backup-${db}.dump"
done

Also back up object storage (MinIO) — mc mirror to a backup bucket — and your configuration (.env / Helm values, stored in your secrets manager).

Restore

A note on “errors ignored on restore” — this is expected, not a failure. On AnvilBase’s Postgres image the database role AnvilBase connects as is demoted from superuser (by the image’s bootstrap). pg_restore therefore cannot recreate a handful of superuser-only platform objects — notably the anvilbase_rls_default event trigger, and occasionally an extension — and prints pg_restore: warning: errors ignored on restore: N, exiting non-zero. All of your tenant data, tables, indexes, and vault.secrets still restore correctly. Those skipped platform objects are re-provisioned by AnvilBase separately (the RLS event trigger is created from template1 on the next project provisioning / migration pass), so they are not a restore failure. The API/CLI/console restore (and the automated restore-verification above) now treat this benign-skip case as success and log the ignored count for visibility; a genuine fatal restore (corrupt archive, connection failure, disk full, or a data-COPY failure that drops table rows) still fails loudly.

Whole platform

docker compose stop
docker compose up -d postgres && sleep 10

# platform DB
docker compose exec -T postgres pg_restore -U postgres --create --clean --if-exists -d postgres < backup-platform.dump

# project DBs
for dump in backup-platform_*.dump; do
  docker compose exec -T postgres pg_restore -U postgres --create --clean --if-exists -d postgres < "$dump"
done

docker compose up -d
curl -s http://localhost:39001/health/services | jq .

One project database

PROJECT_ID="<uuid>"
DB="platform_$(echo "$PROJECT_ID" | tr '-' '_')"
docker compose exec postgres psql -U postgres \
  -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='${DB}'"
docker compose exec postgres dropdb -U postgres "$DB"
docker compose exec -T postgres pg_restore -U postgres --create -d postgres < "backup-${DB}.dump"

Storage

docker compose exec minio mc mirror backup/anvilbase-backups/ local/
# recreate any missing project buckets
docker compose exec control-plane /app/anvilbase storage repair

Recommended schedule

Component	Frequency	Retention
Platform DB	every 6 hours	30 days
Project DBs	daily	14 days
Object storage	daily (incremental)	30 days
Configuration	on every change	90 days

For project databases, prefer the built-in scheduled backups + retention above (no cron container to operate). For the platform database and the whole-deployment dumps, automate with host cron:

# Platform DB every 6 hours
0 */6 * * * docker compose exec -T postgres pg_dump -U postgres -Fc anvilbase_platform > /backups/platform-$(date +\%Y\%m\%d-\%H\%M).dump

Verify your backups

A backup you haven’t restored is a hope, not a backup. The fastest path is to turn on automated restore verification (verify_enabled) so every project’s newest dump is restored into a scratch DB and sanity-checked nightly, with failures surfaced on /health/services.

For a full-stack disaster-recovery drill, also periodically:

Restore the latest dump to a staging stack.
Confirm /health/services is green and /api/v1/projects returns data.
Test an auth login and a storage upload/download.
Confirm the audit log records the recovery.

Going further

For continuous protection and restore-to-any-second, enable Point-in-Time Recovery. For full failure scenarios (database loss, storage loss, corrupted project), see Disaster Recovery.

Next: Point-in-Time Recovery.