Scaling
An instance runs as N identical API replicas and M worker replicas behind any load balancer, with no sticky sessions. Any replica can serve any request and can be stopped at any moment. Scaling up means adding replicas; nothing else changes.
Where state lives
Section titled “Where state lives”| Store | Holds | If it is lost |
|---|---|---|
| Postgres | Everything that must survive: users, sessions, plugin data, outboxes, job claims, idempotency results | It is the source of truth: back it up |
| Redis | Short-lived shared state: rate limits, presence, SSO state, cache-bus and socket pub/sub. Every key has a TTL | Presence resets, an SSO sign-in in progress restarts; no data lost |
| Temporal | Long-running work: workflows, schedules, retries | Its own persistence (on the instance’s Postgres) |
| S3 | File bytes; clients move them directly on signed URLs | Back it up |
| Process | Small caches only, each with a TTL and cleared on every replica by the cache bus | Nothing: rebuilt on demand |
How replicas agree
Section titled “How replicas agree”- Sessions and roles: every request checks its session in Postgres. A sign-out or role change clears the short check cache on every replica and closes or rechecks that person’s sockets everywhere.
- Rate limits: one counter per client in Redis, for the whole instance.
TRUST_PROXYmakes the client’s address the real one behind the balancer. - Outboxes: workflow starts and events are written in the business change’s transaction. Every
replica dispatches, and rows are leased (
SKIP LOCKED), so each goes out once. - Sockets: WebSocket only, so no stickiness. The Redis adapter carries events to sockets on every replica.
- Periodic jobs: each job is claimed in Postgres once per period, whatever the replica count.
- Server-side events: one delivery row per subscriber, claimed by worker replicas, each retried on its own.
- Shutdown: on SIGTERM a replica answers not-ready, moves its sockets off, lets requests and Activities finish within a grace period, then closes its connections. Rolling updates lose nothing.
The rules, for core and for your plugin
Section titled “The rules, for core and for your plugin”- Nothing that matters in process memory.
- Every in-process cache has a short TTL (core’s are also cleared over the cache bus).
- Redis is only for short-lived state, always with a TTL.
- No timers for periodic work: declare a schedule.
- Every side effect is idempotent (keys, workflow ids, event ids,
@Idempotent()). - Your code can be stopped at any moment: long work belongs in workflows.
- Configuration only from the environment (plugin configuration and secrets come from core).
The review checklist checks these on every plugin merge request.
How it is tested
Section titled “How it is tested”- Backend e2e: each mechanism is tested against real Postgres, Redis and Temporal. Some suites run two app instances in one process. The whole suite also runs through PgBouncer.
pnpm --filter backend scale:test: the production image as 3 API and 2 worker replicas behind nginx, tested from the outside.pnpm --filter backend scale:chaos: the same tests in rounds while a random API replica restarts every 12 s. Last run: 120 of 120 tests passed across 7 restarts.
The full reference, with settings, metrics and current limits, is How the backend scales in the
developer docs (apps/documentation/docs/backend/scaling.md).