Monitoring

AnvilBase exposes health endpoints and structured logs you can wire into your observability stack. This page covers what to watch and how.

Health endpoints

Endpoint	Purpose	Response
`GET /health`	liveness	`{"status":"ok"}`
`GET /health/services`	per-service readiness	`{"status":"ok","services":{...},"warnings":[...]}`

/health/services reports each backing service and a warnings[] array. A warning means “still working, but degraded” — investigate, don’t panic. The key warning to alert on is the rate-limiter fallback (below).

curl -s https://<host>/health/services | jq .

Point your uptime monitor (UptimeRobot, Better Uptime, a k8s probe) at /health, and a deeper check at /health/services.

Docker health & logs

# Container health
docker compose ps
docker inspect --format='{{json .State.Health}}' anvilbase-control-plane | jq

# Structured JSON logs
docker compose logs -f --tail 100
docker compose logs -f control-plane auth webhooks

# Filter errors with jq
docker compose logs control-plane --no-log-prefix 2>&1 \
  | jq -r 'select(.level=="ERROR") | "\(.timestamp) \(.message)"'

Ship logs to Loki/Elastic/CloudWatch for retention and search.

Metrics (`/metrics`)

The control plane and the webhooks service expose a Prometheus /metrics endpoint (text exposition, OpenMetrics-compatible).

Exposure model (read this first)

/metrics is NOT a public surface. Traefik only routes /api/v1, /health, and /v1 to the control plane (docker/traefik/dynamic/routing.yml), so /metrics is reachable only on the internal container network — exactly where a Prometheus scraper sits. The webhooks service is internal-only and never Traefik-routed at all. The exposition carries no per-tenant data: every label is a bounded dimension (matched route pattern, HTTP method/status, or a closed op/worker/outcome enum) — never a project id, API key, user, raw path with ids substituted, or any secret. Per-project usage time-series is a separate, admin-scoped API (the usage metering substrate), not this surface.

Defence-in-depth: set ANVILBASE_METRICS_TOKEN on the control plane and webhooks to additionally require Authorization: Bearer <token> on /metrics (add the matching authorization: block to your scrape config). Unset → the standard “scrape unauthenticated on a private network” posture.

Metric families

Metric	Type	Labels	Service
`http_requests_total`	counter	`method`, `route`, `status`	control-plane, webhooks
`http_request_duration_seconds`	histogram	`method`, `route`	control-plane, webhooks
`db_pool_connections`	gauge	`pool` (`platform`\|`projects`), `state` (`in_use`\|`idle`\|`waiting`\|`max`)	control-plane
`db_project_pools`	gauge	— (count of live per-project REST pools — i.e. projects not scaled to zero)	control-plane
`cache_warm_connections`	gauge	— (count of live per-project Valkey cache connections — the cache-side scale-to-zero warm count)	control-plane
`data_plane_ops_total`	counter	`subsystem` (`queue`\|`cache`\|`cron`), `op`	control-plane
`worker_runs_total`	counter	`worker`, `outcome` (`success`\|`failure`)	control-plane
`usage_flushed_total`	counter	— (per-project usage deltas UPSERTed; the breakdown lives in `project_usage`, never a label)	control-plane
`usage_storage_sampled_total`	counter	— (per-project storage samples written)	control-plane
`webhook_deliveries_total`	counter	`outcome` (`delivered`\|`retrying`\|`dead_letter`\|`failed`)	webhooks
`webhook_send_duration_seconds`	histogram	—	webhooks

route is the matched route pattern (e.g. /v1/queue/{project_id}/queues), so the live project id never reaches a label — cardinality stays bounded. db_pool_connections{pool="projects"} is an aggregate across every per-project REST pool (never one series per project).

Useful expressions:

rate(http_requests_total[5m])                                       # request rate
rate(http_requests_total{status=~"5.."}[5m])                        # error rate
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # p99 latency
db_pool_connections{pool="platform",state="in_use"}
  / db_pool_connections{pool="platform",state="max"}                # platform pool saturation
rate(webhook_deliveries_total{outcome="dead_letter"}[15m])          # webhook DLQ rate
rate(worker_runs_total{outcome="failure"}[15m])                     # background-worker failures

The bundled Grafana dashboard (Mode 1 below) ships exactly this — request rate, 5xx rate, p50/p95/p99 latency, top routes, DB-pool saturation, data-plane ops, worker-run failures, access-log flush/drop, usage flush, and webhook delivery outcomes — provisioned and ready. Pair with the log drain (below) for logs.

There are two modes to consume AnvilBase metrics + logs. Pick whichever fits your network topology — they are not mutually exclusive (the OTLP push and the scrape surface are fed from the same in-process recorder, so you can run both).

Mode 1 — sidecar / self-contained. Run the bundled Prometheus + Grafana in the observability Compose profile. Grafana comes up with the datasource and the AnvilBase dashboard already provisioned — no manual import. Best when you want metrics on this box with nothing external.
Mode 2 — connect an existing central stack. You already run a single central observability stack (Prometheus/Thanos/Mimir + Loki + Grafana, e.g. Observatorium). Because a central stack usually can’t scrape into a deployment’s network, AnvilBase pushes: set OTEL_EXPORTER_OTLP_ENDPOINT to export metrics + traces over OTLP, and/or run a scrape→remote_write agent for the Prometheus path, plus a logs drain.

Mode 1 — sidecar Grafana + Prometheus (self-contained)

Docker Compose ships a profile-gated Prometheus + Grafana sidecar — both are out of the default up so the base stack stays lean and an existing deployment is never forced to run them:

docker compose --profile observability up prometheus grafana
# Prometheus UI:  http://localhost:39090   (PROMETHEUS_PORT)
# Grafana:        http://localhost:39091   (GRAFANA_PORT, default login admin/admin)

Prometheus scrapes control-plane:3001/metrics and webhooks:3003/metrics on the internal network (docker/prometheus/prometheus.yml). Grafana auto-provisions from docker/grafana/:

the Prometheus datasource (provisioning/datasources/prometheus.yml), and
the AnvilBase — Overview dashboard (dashboards/anvilbase-overview.json, loaded via provisioning/dashboards/dashboards.yml).

Open Grafana → the AnvilBase folder → AnvilBase — Overview. No import step. Set GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD (and a real PROMETHEUS_PORT / GRAFANA_PORT) in .env for anything but a local box. Replace the bundled Prometheus/Grafana with your own or a managed scraper in production — the dashboard JSON is portable (drop it into any Grafana with a Prometheus datasource).

Mode 2 — connect an existing central stack (OTLP push / remote_write)

When a central observability stack can’t scrape into your network (it lives elsewhere, behind a firewall, multi-tenant — e.g. Observatorium), AnvilBase pushes out. There are two push paths; they coexist with each other and with the scrape surface (all fed from one in-process recorder).

2a — OTLP push (recommended, zero extra infra)

Set OTEL_EXPORTER_OTLP_ENDPOINT on the control-plane + webhooks services and they export both traces and metrics over OTLP/HTTP to that endpoint — the same metric families as the scrape surface, on a 60s periodic push. Add OTEL_EXPORTER_OTLP_HEADERS for auth (Bearer token, multi-tenant X-Scope-OrgID). Unset → push is off entirely (scrape-only, zero cost); it is the same single env gate that already controls trace export.

# Push metrics + traces to a central OTLP-capable stack — your Observatorium
# OTel collector, a standalone OpenTelemetry Collector, or Grafana Alloy. These
# are PLACEHOLDERS — substitute your endpoint + token; never commit a real one.
OTEL_EXPORTER_OTLP_ENDPOINT=https://<observatorium-otlp-host>/otlp
# Comma-separated; URL-encode values (%20 = space). Auth + tenant routing:
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<OTLP_TOKEN>,X-Scope-OrgID=<your-tenant-id>

The central collector forwards to Prometheus-compatible storage (Thanos / Mimir / Cortex), where your Grafana queries the same http_requests_total / http_request_duration_seconds / db_pool_connections / webhook_deliveries_total families. No /metrics route needs to be reachable from outside. Import the same dashboard (docker/grafana/dashboards/anvilbase-overview.json) into your central Grafana — it queries by metric name, so it works against the pushed series too.

2b — scrape → `remote_write` (Prometheus path to Thanos-receive)

If your central stack is Prometheus-native and you prefer remote_write over OTLP, run an agent (a local Prometheus, Grafana Alloy, or an OpenTelemetry Collector) that scrapes AnvilBase /metrics locally and remote_writes to a central Thanos-receive / Mimir / Cortex endpoint. The standard path — operators expect it, and AnvilBase needs no Rust-side remote_write client.

Local Prometheus: uncomment the remote_write: block in docker/prometheus/prometheus.yml (shipped commented, with placeholder URL + X-Scope-OrgID tenant header + Bearer token) and run the observability profile’s Prometheus.
OpenTelemetry Collector: a ready reference config lives at docker/monitoring/otel-collector.example.yaml — it can scrape /metrics (or receive the OTLP push from 2a), prometheusremotewrite to Thanos-receive, and drain container logs to Loki. Every endpoint/token in it is a placeholder.

Concrete example — connect your existing Grafana / Observatorium

Say your central stack is Observatorium with an OTLP ingest, a Thanos-receive remote-write endpoint, and a Loki logs plane, tenant team-anvil:

# (A) Metrics + traces via OTLP push — set on control-plane + webhooks:
OTEL_EXPORTER_OTLP_ENDPOINT=https://obs.example.com/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20$OBS_TOKEN,X-Scope-OrgID=team-anvil

# (B) — OR — Prometheus remote_write to Thanos-receive
#      (docker/prometheus/prometheus.yml):
remote_write:
  - url: "https://obs.example.com/api/v1/receive"
    headers: { X-Scope-OrgID: "team-anvil" }
    authorization: { type: Bearer, credentials: "<REMOTE_WRITE_TOKEN>" }

# (C) Logs → Loki via the agent (see the log-drain section + the OTel collector
#      example). Vector/Fluent-Bit/OTel-Collector all work; set the Loki
#      endpoint + X-Scope-OrgID tenant header.
sinks:
  loki:
    type: loki
    endpoint: https://obs.example.com/loki/api/v1/push
    request: { headers: { X-Scope-OrgID: "team-anvil" } }
    auth: { strategy: bearer, token: "<LOKI_TOKEN>" }

Then point your existing Grafana at the Observatorium datasources and import docker/grafana/dashboards/anvilbase-overview.json. All placeholders — swap in your real hosts/tokens; nothing here depends on a path outside this repo.

Realtime & auth metrics (follow-up)

The realtime (Phoenix) and auth (Bun/Hono) services do not yet expose /metrics — that is a precise, scoped follow-up, intentionally not half-wired here:

Realtime: add telemetry_metrics_prometheus (or PromEx) to the Phoenix endpoint and expose a /metrics route emitting the Phoenix/Bandit telemetry (connected sockets, channel joins, DB-listener counts). Add the same prometheus.io/* pod annotations + a compose scrape target.
Auth: add a Hono /metrics route (a small prom-client registry or a hand-rolled counter) exposing request count/latency and sign-in/sign-up counters, with the same annotations + scrape target.

Until then, watch realtime/auth via /health, the structured logs (below), and the auth-event audit feed.

Per-project logs API

Where /metrics gives you the aggregate view (no per-tenant series — that would be unbounded cardinality and a tenant-data leak), the per-project logs API gives the console (and you) a queryable, tenant-scoped view of a single project’s recent activity. It pairs with metrics: metrics tell you the platform’s overall request rate/latency; the logs API tells you what one project did.

GET /api/v1/projects/{id}/logs?source=function|access|all&limit=&before=

Admin/console surface (gated by the management-plane auth + RBAC; classified as a read, so a read PAT or viewer can call it). It reads two per-project stores and returns them newest-first, merged:

`source`	Store	What it captures
`access`	`proxy_access_logs`	One row per per-tenant data-plane request through the `/v1/...` proxy (REST, storage, auth, functions, realtime, queue, cache, cron, graphql, eidas): `method`, the matched route pattern, `status`, `duration_ms`, `created_at`.
`function`	`edge_function_logs`	One row per edge-function invocation: `function_name`, `method`, `status`, `duration_ms`, a short `message`, `created_at`.
`all`	both	The two streams merged by `created_at` (the default).

Parameters

source — function, access, or all (default all).
limit — page size, 1..=500 (default 50).
before — an opaque keyset cursor (a compound (created_at, id) token); returns only rows older than it. The response carries next_before when the page was full — pass that value straight back as before to walk to the next (older) page. Treat it as opaque (a malformed cursor is a clean 400). It’s compound, not a bare timestamp, so paging never skips rows that share a created_at (e.g. a whole capture batch written in one transaction).

# Newest 50 across both sources
curl -s -H "Authorization: Bearer $ANVILBASE_PAT" \
  "https://<host>/api/v1/projects/$PID/logs" | jq .

# Just the data-plane access logs, 100 at a time
curl -s -H "Authorization: Bearer $ANVILBASE_PAT" \
  "https://<host>/api/v1/projects/$PID/logs?source=access&limit=100" | jq '.logs[]'

What is captured — and what is deliberately NOT

The stores hold telemetry shape only. Specifically captured: HTTP method, the matched route pattern (e.g. /v1/cache/{project_id}/set — never the raw path with the live id substituted in), status, duration, and (for functions) the function name + a short non-sensitive message.

Never captured or returned: request/response bodies, the Authorization header, API keys, JWTs, the query string (which can carry a realtime ?apikey=/?token= credential), or any other secret. A project’s /logs returns only its own rows (every query is WHERE project_id = $1) — there is no cross-project read path.

Capture cost (hot-path safety)

Access-log capture is cheap and non-blocking. The proxy records a request by pushing one entry into a bounded in-memory channel (a single non-blocking try_send) and returns immediately — it never does a synchronous DB write on the request path. A background flusher drains the channel and batch-inserts the rows. If a burst outpaces the flusher, the channel fills and capture drops the overflow (surfaced as access_log_dropped_total on /metrics) rather than blocking the request or growing memory unbounded.

Retention (bounded growth)

Both stores are per-project ring buffers with a hard upper bound — they never grow unbounded. An AFTER INSERT trigger trims a project’s rows past the newest ~2000 (proxy_access_logs) / ~1000 (edge_function_logs). The trim is probabilistically gated (only a small fraction of inserts run the trim) so the common insert path stays cheap; amortized, each project stays near its cap within a small bursty slack. No external sweeper is required.

Usage metering

Where the logs API records individual events and /metrics gives the aggregate rate, the usage metering time-series records each project’s usage over time — the history the access-log ring buffer and the point-in-time resource limits don’t keep.

GET /api/v1/projects/{id}/usage?from=&to=&limit=

Admin/console surface (gated by the management-plane auth + RBAC; classified as a read). It returns the project’s hourly usage buckets, newest-first.

What is captured

Per-tenant data-plane requests through the /v1/... proxy accrue four counters, plus a periodically-sampled storage gauge, into the project’s current hourly bucket (bucket_start = date_trunc('hour', …)):

Field	Kind	Meaning
`requests`	counter (summed)	per-tenant data-plane requests in the hour
`bytes_in`	counter (summed)	request bytes (from `Content-Length`)
`bytes_out`	counter (summed)	response bytes (from `Content-Length`)
`function_invocations`	counter (summed)	edge-function invocations in the hour
`storage_bytes`	gauge (last sample)	sampled project DB size (`pg_database_size`)

Within an hour the counters accumulate across the metering flusher’s flush ticks (a delta is added, never overwritten); storage_bytes is a gauge — the storage sampler sets it to the latest sampled value.

# Last 7 days of hourly usage
curl -s -H "Authorization: Bearer $ANVILBASE_PAT" \
  "https://<host>/api/v1/projects/$PID/usage" | jq '.usage[]'

Parameters: optional from/to (RFC3339) bound the bucket_start window (a malformed value is a clean 400); limit bounds the number of buckets (1..=5000, default 168).

Capture cost (hot-path safety)

Usage capture is even cheaper than the access log: the proxy bumps in-memory per-project atomic counters (a lock-cheap fetch_add) — it does no DB write and no .await on the request path, and it never buffers or measures a streamed body (byte counts come from cheap Content-Length header reads; an unknown size contributes 0). A background flusher periodically drains the counters and UPSERTs the accumulated deltas into the current hourly bucket; on a much slower cadence it samples each active project’s DB size off the request path entirely. A flush or sample error is logged and skipped — metering never crashes the server or blocks a request.

What is NOT captured

A usage bucket carries only aggregate numbers + the bucket timestamp — no request/response body, header, API key, JWT, raw path, or any per-request detail. A project’s /usage returns only its own buckets (WHERE project_id = $1) — there is no cross-project read path, and the Prometheus surface stays aggregate (the per-tenant breakdown lives in this table, never a metric label).

Retention (bounded growth)

The table is a low-cardinality time-series — one row per project per hour. An AFTER INSERT trigger prunes buckets older than 90 days for the project, so it stays bounded without an external sweeper. Because the metering flusher UPSERTs at most once per project per flush tick, the trigger fires rarely and the prune is a single cheap ranged delete.

Audit & auth-event retention

The platform audit_log table (every administrative action + every auth event) and the internal auth-event PGMQ transport are append-only, so without pruning they grow forever. AnvilBase ships a retention prune for both — a nightly schedule plus an on-demand operator path.

Nightly schedule: pg_cron jobs in the platform database (anvilbase_platform) — the same scheduler the webhook DLQ purge uses. Two jobs run with a fixed 90-day window:
- prune_audit_log_nightly (03:30 UTC) → SELECT prune_audit_log(90) deletes audit_log rows older than 90 days.
- prune_auth_event_archive_nightly (03:40 UTC) → SELECT prune_auth_event_archive(90) deletes archived auth events (pgmq.a_auth_events) older than 90 days and genuine poison messages on the queue (pgmq.q_auth_events) — rows that are both older than the window and have failed redelivery many times (read_ct >= 5). An undrained backlog of real, unprocessed events (read_ct = 0) is never deleted, so a recovered consumer can still process it — auth durability is preserved.
On-demand prune: POST /api/v1/admin/audit/prune (and anvilbase audit prune [--days N]) runs both prunes immediately for incident cleanup or a tightened window. It defaults to ANVILBASE_AUDIT_RETENTION_DAYS (default 90; values <1 fall back to 90) when --days / ?days= is omitted. This env var controls the on-demand default only — it does not change the nightly schedule.
Non-fatal & bounded: each prune is a single ranged DELETE; a scheduled failure is logged by pg_cron and retried the next night. The window keeps both stores from unbounded growth between exports.
No pg_cron? On a stock Postgres without pg_cron, the prune functions are still installed (migration 000026) — only the auto-schedule is skipped. Use the admin endpoint above, or invoke the functions from your own scheduler:
```
-- as the anvilbase role, against anvilbase_platform
SELECT prune_audit_log(90);
SELECT prune_auth_event_archive(90);
```

Change the nightly window: repoint the cron job (the env var only affects the on-demand path):

UPDATE cron.job SET command = 'SELECT prune_audit_log(30)'
 WHERE jobname = 'prune_audit_log_nightly';

Export before rows age out. The audit_log retention window is a hard floor on how far back the in-database log reaches. For long-term / compliance retention, export to immutable off-box storage on a schedule — see Audit Logs → Export.

Log drain

Every service logs structured JSON to stdout (the Rust services via tracing-subscriber’s JSON layer; auth/realtime via their own structured loggers). The container runtime captures stdout, so you drain logs by pointing a collector at the container logs — no app-side change needed. This is the logs half of Mode 2 (connect an existing central stack): the same Vector / Fluent Bit / OpenTelemetry Collector that ships your metrics also ships your logs to the central store. For a multi-tenant store (Observatorium / Loki / Mimir) add your tenant header (X-Scope-OrgID) to the sink (shown in the Mode 2 example above and the OTel collector reference at docker/monitoring/otel-collector.example.yaml).

Vector (example)

vector.yaml — tail the Docker JSON log files, parse the embedded message, and forward to your sink (Loki shown):

sources:
  anvilbase_containers:
    type: docker_logs
    include_containers:
      - anvilbase-control-plane
      - anvilbase-webhooks
      - anvilbase-auth
      - anvilbase-realtime

transforms:
  parse_json:
    type: remap
    inputs: [anvilbase_containers]
    source: |
      # The Rust services emit JSON on stdout; parse it so `level`, `message`,
      # `timestamp`, and span fields become first-class fields. Non-JSON lines
      # pass through untouched.
      structured = parse_json(.message) ?? {}
      . = merge(., structured)

sinks:
  loki:
    type: loki
    inputs: [parse_json]
    endpoint: http://loki:3100
    labels:
      service: "{{ container_name }}"
      level: "{{ level }}"
    encoding:
      codec: json

Fluent Bit (example)

fluent-bit.conf — same idea via the Docker log driver / tail input:

[INPUT]
    Name              tail
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    Tag               anvilbase.*

[FILTER]
    Name              parser
    Match             anvilbase.*
    Key_Name          log
    Parser            json
    Reserve_Data      On

[OUTPUT]
    Name              loki
    Match             anvilbase.*
    host              loki
    port              3100
    labels            job=anvilbase, level=$level

Both forward the already-structured JSON so level, timestamp, message, and tracing span fields (e.g. anvilbase.project_id on request spans) stay queryable in your log store. Ship to Loki/Elastic/CloudWatch for retention and search.

Alerts that matter

Alert	Condition	Why
Service down	`up == 0` for 1m	hard outage
High error rate	5xx / total > 5% for 5m	regression or dependency failure
Auth failure spike	failures > 10% in 5m	possible brute force — check IPs
DB connections saturated	active / max > 90%	raise pool or scale (Scaling)
Storage near quota	usage > 80–90%	notify user / raise limit
Cert expiring	< 30 days	rotate before outage
Rate limiter degraded	`warnings[]` contains `rate_limit_degraded_fallback_to_in_memory` while replicas > 1	rate limits not shared across replicas — Valkey unreachable (Scaling)
Backup restore-verify failed	`warnings[]` contains `backup_restore_verification_failed`	a project’s newest backup failed automated restore verification — the backup may be unrestorable (Backups & Restore)

The rate-limiter warning is the single most important multi-replica alert: pin on that exact string. The backup restore-verification warning means a backup you may need is not provably restorable — treat it as data-protection-critical.

Audit log as a monitoring source

The audit log is a security telemetry feed. Alert on sensitive actions:

# Failed auth attempts per project in the last hour (export + analyze)
curl ".../api/v1/admin/audit/export?action=auth.signin.failed&from=...&to=..." -o fails.csv

Forward the audit export into your SIEM and alert on secret.reveal.service_role, project.delete, and platform-user creation.

External uptime check (quick win)

*/5 * * * * curl -sf https://<host>/health || echo "AnvilBase DOWN" | mail -s "Alert" ops@example.com

Recommended tooling

Uptime: UptimeRobot / Better Uptime, plus k8s probes.
Metrics: Prometheus + Grafana.
Logs: Loki (or Elastic / CloudWatch).
Paging: PagerDuty / Opsgenie.

Next: Scaling.

Monitoring

Health endpoints

Docker health & logs

Metrics (/metrics)

Exposure model (read this first)

Metric families

Mode 1 — sidecar Grafana + Prometheus (self-contained)

Mode 2 — connect an existing central stack (OTLP push / remote_write)

2a — OTLP push (recommended, zero extra infra)

2b — scrape → remote_write (Prometheus path to Thanos-receive)

Concrete example — connect your existing Grafana / Observatorium

Realtime & auth metrics (follow-up)

Per-project logs API

What is captured — and what is deliberately NOT

Capture cost (hot-path safety)

Retention (bounded growth)

Usage metering

Capture cost (hot-path safety)

What is NOT captured

Retention (bounded growth)

Audit & auth-event retention

Log drain

Vector (example)

Fluent Bit (example)

Alerts that matter

Audit log as a monitoring source

External uptime check (quick win)

Recommended tooling

Metrics (`/metrics`)

2b — scrape → `remote_write` (Prometheus path to Thanos-receive)