Disaster Recovery

This is the break-glass runbook. Keep it accessible off the AnvilBase host. Recovery priority follows where state lives:

1. PostgreSQL — all state lives here (recover first).
2. Storage (MinIO) — user-uploaded files.
3. Control plane — stateless; rebuild from image.
4. Auth — stateless; sessions live in Postgres.
5. Everything else — stateless services.

Prerequisite: working backups and, ideally, PITR. Test these before you need them.

Scenario: complete database loss

From logical backup (pg_dump)

docker compose stop
docker compose up -d postgres && sleep 10

# platform DB
docker compose exec -T postgres pg_restore -U postgres --create --clean --if-exists -d postgres < backup-platform.dump

# each project DB
for dump in backup-platform_*.dump; do
  docker compose exec -T postgres pg_restore -U postgres --create --clean --if-exists -d postgres < "$dump"
done

docker compose up -d
curl -s http://localhost:39001/health/services | jq .
curl -s http://localhost:39001/api/v1/projects -H "Authorization: Bearer $ANVILBASE_TOKEN"

From a volume snapshot

docker compose stop postgres
# restore the Postgres volume from your provider snapshot (EBS/Disk/etc.)
docker compose up -d postgres && sleep 10
docker compose up -d
docker compose exec postgres psql -U postgres -d anvilbase_platform -c "SELECT count(*) FROM projects"

To a specific point in time

If PITR is enabled, restore to any timestamp in the WAL window — see Point-in-Time Recovery → Trigger a restore. This is the lowest-RPO option.

Scenario: storage (MinIO) loss

# restore objects from your backup bucket
docker compose exec minio mc mirror backup/anvilbase-backups/ local/

# recreate any missing per-project buckets
docker compose exec control-plane /app/anvilbase storage repair

If the MinIO volume is gone and no backup exists, the objects are lost — which is why daily storage backups are on the checklist.

Database dumps survive a MinIO-volume loss when the offsite copy is configured (BACKUP_OFFSITE_* — see Backups & Restore → Offsite copy): every completed project backup is asynchronously mirrored to a secondary S3 target, so you can pull the dumps back from there and restore each project DB.

Scenario: single service failure

docker compose ps
docker compose logs <service> --tail 50
docker compose restart <service>

# if the image is corrupted
docker compose build --no-cache <service>
docker compose up -d <service>

Scenario: corrupted project database

PROJECT_ID="<uuid>"
DB="platform_$(echo "$PROJECT_ID" | tr '-' '_')"

docker compose exec postgres psql -U postgres \
  -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='${DB}'"
docker compose exec postgres dropdb -U postgres "$DB"
docker compose exec -T postgres pg_restore -U postgres --create -d postgres < "backup-${DB}.dump"

docker compose exec postgres psql -U postgres -d "$DB" \
  -c "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'"

Only that project is affected; others keep running (the benefit of database-per-project isolation).

Post-recovery verification

• curl /health/services — all services healthy.
• curl /api/v1/projects — returns data.
• Console reachable (:39004).
• Auth login/signup works.
• Storage upload/download works.
• Audit log shows the recovery event.

RTO / RPO planning

Strategy	Typical RPO	Typical RTO
Daily logical backups	up to 24h	restore time (minutes–hours)
6-hourly platform backups	up to 6h	restore time
PITR (wal-g)	seconds–1 min	restore + replay time
Volume snapshots	snapshot interval	fast volume swap

Combine: PITR for low RPO, periodic logical dumps for portability and a guaranteed known-good baseline. Rehearse recovery on staging so your RTO is real, not theoretical.

Next: CLI Reference.