Edge Functions Runtime & Isolation

This page documents the isolation model and lifecycle of the edge functions runtime: how AnvilBase keeps one tenant’s functions from reading, crashing, or interfering with another’s.

Isolation model: one process per project

The deno service is a minimal supervisor (docker/deno/supervisor.ts) listening on :8082. It runs no function code itself. For each project that receives an invocation, it spawns at most one worker subprocess (docker/deno/worker.ts) and proxies the request to it over loopback.

Each worker is locked down by the Deno permission model to its own project:

deno run --allow-net --allow-env \
  --allow-read=/functions/<project_id>,/tmp/anvilbase-workers/<project_id> \
  --allow-write=/functions/<project_id>,/tmp/anvilbase-workers/<project_id> \
  worker.ts
# env: DENO_DIR=/tmp/anvilbase-workers/<project_id>  (per-worker module cache)

The second path is the worker’s own isolated Deno module cache (DENO_DIR), so a function’s remote / npm: / jsr: imports resolve deterministically without sharing a writable cache across tenants. A worker still cannot reach any other tenant’s source or cache.

This buys two concrete guarantees:

• Filesystem isolation. A function in project A physically cannot read or write another tenant’s source on the shared /functions volume. A call to Deno.readTextFile("/functions/<other-project>/fn/index.ts") raises NotCapable (Deno 2.x; PermissionDenied on older runtimes) — it never returns the file contents. This is enforced by the OS/Deno sandbox, not by convention.
• Crash isolation. A worker is a separate OS process and event loop. A function that calls Deno.exit(), panics, or exhausts memory kills only its own project’s worker. Other projects’ workers keep serving. The supervisor restarts the crashed project’s worker on its next request (behind a backoff — see below).

Per-invocation isolation is now opt-in

By default the isolation unit is an OS process per project (the pooled, warm-worker model above): concurrent invocations of the same project share that project’s worker process and module cache. Cross-project isolation is complete. Intra-project, per-invocation isolation — a brand-new V8 isolate for every request, so no module-level / global state leaks between calls — is now available as an opt-in mode (ANVILBASE_EDGE_ISOLATION=per-invocation, see Isolation modes below). It is off by default so the running stack is byte-unaffected. Per-invocation secrets are carried on the request object (req.anvilbase) in both modes, never in process-global state.

Isolation modes: per-project vs per-invocation

The worker lifecycle is selected by the ANVILBASE_EDGE_ISOLATION env var on the deno service:

Mode	Worker lifecycle	State between invocations	Default
per-project	One pooled worker per project, reused warm across invocations	A module-level / global variable set on one call may be observed by a later call to the same project (shared process).	✅ default (unset → this)
per-invocation	A fresh worker subprocess per request, torn down after the response	None. Every request gets a fresh V8 isolate, fresh globals, and a fresh module registry — a variable mutated on invocation #1 is never observed on #2.	opt-in

Tenant isolation holds in both modes. Per-invocation does not weaken the cross-tenant boundary: each fresh worker is spawned with the same Deno permission scoping (--allow-read=/functions/<project_id>,...), the same per-project secret injection, and the same isolated module cache as the pooled worker. A function in project A still gets NotCapable / PermissionDenied if it tries to read project B’s source — the only thing that changes is when the worker dies (after one request vs. when it’s evicted/idle/crashes).

Opting into per-invocation

# docker-compose.yml → deno service (or .env)
ANVILBASE_EDGE_ISOLATION: "per-invocation"   # default: "per-project"

Any unrecognized value falls back to the safe per-project default. In per-invocation mode GET /health reports "mode":"per-invocation" and "workers":0 between requests (workers are ephemeral, never pooled). The durable-deploy + lazy-rehydration flow, the invoke HTTP contract, CORS preflight, mTLS, and verify_jwt (control-plane side) are all preserved exactly as in per-project mode.

Cold-start trade-off

A fresh subprocess per request means every invocation pays a cold-start cost (spawn + V8 init + module load), not just the first call to an idle project. This is the correctness/isolation-first implementation; a warm-isolate pool (pre-spawned, recycled isolates that still reset per-request state) is an explicit follow-up optimization and is not built yet. Use per-invocation where strict per-call statelessness matters more than latency (e.g. defense-in-depth for untrusted multi-tenant logic); keep the default per-project for latency-sensitive workloads.

The per-invocation lifecycle is: spawn a fresh worker → route the single request to it → buffer its response → tear the worker down (SIGTERM, then SIGKILL after a 5 s grace) before returning. Teardown is awaited so that by the time the caller sees the response there is no orphaned subprocess — a guarantee exercised by the test suite (workers:0 after every response).

What a function sees (unchanged)

The runtime contract is identical to before isolation landed — deployed functions need zero changes. The control plane injects per-invocation project context as X-AnvilBase-* headers, which the worker exposes as req.anvilbase. See Overview and Writing & Deploying.

The worker pool

The supervisor manages worker processes with these policies (all tunable via env on the deno service):

These policies apply to the per-project (pooled) mode. In per-invocation mode workers are ephemeral (one per request, torn down after), so the pool cap, LRU, idle reaping, and warm reuse below do not apply — only the per-request timeout and the spawn/handshake machinery are shared.

Concern	Default	Env var
Isolation mode (worker lifecycle)	per-project	ANVILBASE_EDGE_ISOLATION
Max live workers (per-project mode)	16	DENO_MAX_WORKERS
Eviction past the cap	LRU (SIGTERM, then SIGKILL after 5 s)	—
Idle eviction	after 5 min idle (quiet)	DENO_WORKER_IDLE_MS
Per-request upstream timeout	30 s (matches the proxy budget)	DENO_REQUEST_TIMEOUT_MS
Crash backoff (start → cap)	1 s → ×2 → 30 s	DENO_WORKER_BACKOFF_START_MS, DENO_WORKER_BACKOFF_CAP_MS
Backoff reset after healthy run	60 s	DENO_WORKER_HEALTHY_RESET_MS
Worker start (port handshake) timeout	10 s	DENO_WORKER_START_TIMEOUT_MS

Lifecycle

1. Spawn on demand. The first request for a project spawns its worker. The worker binds an ephemeral loopback port (127.0.0.1:0) and announces it on stdout (ANVILBASE_WORKER_READY <port>). Ephemeral ports are race-free — no port-range bookkeeping, no bind collisions. If the worker dies before announcing, the supervisor treats it as a crash (below) rather than hanging.
2. Warm reuse. Subsequent requests for the same project reuse the live worker. Warm-path overhead is a single loopback hop (~1 ms).
3. Cap & LRU. When DENO_MAX_WORKERS live workers exist and a new project needs one, the least-recently-used worker is evicted (graceful SIGTERM, then SIGKILL after 5 s).
4. Idle reaping. A worker idle longer than DENO_WORKER_IDLE_MS is evicted quietly.
5. Crash & backoff. When a worker crashes, the supervisor restarts it on the next request — but only after an exponential backoff window (1 s, doubling, to a 30 s cap; reset to 1 s after the project has run healthy for a minute). While a project is in backoff, the supervisor returns 503 with a Retry-After header, which the functions proxy passes straight through to the caller.

Health

GET http://deno:8082/health
→ 200 {"status":"ok","workers":<n>,"projects":[<project_id>,...]}

status:"ok" is preserved for the Compose healthcheck, scripts/smoke-test.sh, and the control-plane /health fan-out; workers/projects expose the live pool census.

Rollback hatch

The legacy single-process runtime (bootstrap.ts) — all tenants in one shared Deno process with --allow-read=/functions over the whole volume — is retained for one release as a rollback escape hatch:

# docker-compose.yml → deno service
DENO_SUPERVISOR: "off"   # default is "on" (supervisor mode)

DENO_SUPERVISOR=off has no cross-tenant isolation — any function can read any other project’s source on the shared volume, and one crash takes down every tenant’s functions. Use it only to unblock an incident, and revert.

Tests

The isolation guarantees are exercised by the deno test suite (docker/deno/supervisor_test.ts + docker/deno/per_invocation_test.ts, run in CI via deno test --allow-all docker/deno/), including the cross-tenant filesystem-denial acceptance test, crash-then-recover, LRU and idle eviction, the port-discovery race, and lazy rehydration. The per-invocation suite additionally proves: the default mode is per-project (the pooled path is taken and a warm worker is retained), per-invocation mode does not leak module state between invocations (contrasted against pooled mode, which does retain it), the tenant FS boundary still holds on a fresh per-invocation worker, secrets are injected per invocation, CORS preflight is answered without a spawn, and teardown leaves no orphaned worker (workers:0 after every response).