Architecture

Spillway is one Node process. It serves the JSON API, the admin SPA and the public pages from the same Hono app, against one SQLite file or a Postgres database. There is no queue, no cache server and no separate renderer unless you add one: Redis, Meilisearch and Postgres plug in through environment variables, modules through a sandbox or an in-process loader. This document describes how that process is put together and where those optional pieces attach.

One process

packages/core/src/app.ts (createApp) builds everything without listening, which is also how the tests run it (app.request()):

1. Resolve config from the environment (config.ts), open the database (db/index.ts: SQLite with WAL mode, foreign keys on and a 5 s busy timeout, or a Postgres pool when DATABASE_URL is a postgres:// URL) and run pending migrations.
2. Build the AppContext: config, db, logger, the page cache (an in-process LRU of 500 entries, or Redis), the login rate limiter (10 attempts per 15 minutes per IP), the cached "installed" flag, and the services (AI, modules, SSO and SCIM, the importer, the registry when REGISTRY_HOST=1).
3. Mount middleware and routes in this order:
   • onError: HttpError becomes { error: { code, message, details } } with its status; zod errors become 400 validation_error; anything else is logged and becomes 500. Requests outside /api that accept text/html get a small HTML error page instead.
   • Security headers on every response (X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, X-Frame-Options, a Content Security Policy, HSTS when the site is https; see security.md).
   • Body limits: every request body is bounded before a route reads it (JSON 1 MB, media uploads 25 MB, module archives 64 MB, registry publishes 20 MB, the public form submit and the beacon 256 KB).
   • Session middleware: reads the session cookie, loads the user, sets c.var.user and c.var.permissions. Never rejects; routes decide. Bearer middleware: an API token (Authorization: Bearer spw_…) replaces the cookie user and narrows the permissions to the token's subset.
   • Site middleware (auth/site.ts): resolves the request's site from the Host header (or a permitted ?__site= / X-Spillway-Site override), sets c.var.site and c.var.access, and recomputes c.var.permissions for that site. Workstream scope: X-Spillway-Workstream (or ?workstream=) puts the request inside a branch.
   • Setup redirect: until a user exists, every request that is not /api/*, /admin*, /healthz or /registry/v1/* is redirected to /admin, where the SPA shows the wizard.
   • Redirect middleware: the site's redirects win over the page lookup; the same middleware logs 404s.
   • CSRF check on /api/* for every non-GET/HEAD/OPTIONS request, except bearer requests and the public form, DAM webhook and TMS webhook posts, which are verified in other ways.
   • API routers under /api, one per area, mounted in a fixed order (setup, auth, types, review, bulk, editorial, TMS and translations, entities, forms, DAM and media, site, sites, preferences, sections, kits, alerts, menus, theme, personalization, analytics, users, tokens, redirects, collections, vocabularies, shared blocks, rich text, search, the public render API, AI, brand, modules, SSO, workstreams, the importer, the registry). Unknown /api/* paths give 404 JSON.
   • Then /healthz, POST /mcp, the SSO browser flow (/sso/*), SCIM (/scim/v2), the registry's public API (/registry/v1/*), public media files (/media/:id/:filename), the admin SPA (/admin, /admin/*), module assets (/modules/<name>/*), the SEO files, feeds, /_alerts, /_island, /_beacon, and finally the catch-all page renderer.

server.ts wraps createApp with @hono/node-server, listens on PORT (and HOST, default 0.0.0.0), starts the ticker unless SPILLWAY_TICKER=0, and closes the database on SIGINT/SIGTERM. cli.ts is the spillway binary; its commands are listed in install.md and generated into reference/cli.md.

Layers inside packages/core/src:

The feature areas each have a folder of their own next to these: marketing/ (collections, search indexing, SEO files, redirects), forms/ (definitions, submissions, uploads, CAPTCHA, connectors and the outbound HTTP guard forms/http.ts), review/ (checks, comments, the queue), notifications/ (rules and the outbox), jobs/ (the ticker), events/ (recurrence and occurrences), taxonomy/, shared-blocks/, alerts/, menus/, theme/ (token import and export), personalization/, analytics/, ai/ (providers, tasks, prompts), mcp/, modules/ (manifests, the trusted loader, the Wasm sandbox host, the registry client, safe mode), registry/ (the registry host), kits/ (the installer and dependency planner), sites/ and auth/site.ts (multisite and sections), i18n/ (admin messages, XLIFF, the TMS providers), workstreams/, importer/, sso/, scim/, net/, richtext/ (the sanitizer). Each has its page under docs/.

Hybrid schema

The initial tables (migration 0001_init) are site, user, session, content_type, entity, revision, reference, media, audit_log and _migrations; every later feature adds its own group of tables in a numbered migration under db/migrations/ (the list is in index.ts).

The schema is hybrid: the columns the server filters, sorts or joins on are typed columns, and the user-defined fields are one JSON data column.

• entity has typed columns for type, site_id, locale, status, title, slug, path, author_id, revision pointers and timestamps, with indexes on (type, status), updated_at and a unique index on (site_id, locale, path) where path is not null. Field values live in data (JSON text).
• revision holds a full snapshot (title, data) per save, numbered per entity with a unique (entity_id, number). Rows cascade-delete with their entity.
• reference is a join table filled from reference fields on every save, so "what links here" is a query, not a JSON scan.
• content_type.fields is JSON (FieldDef[]); the rest of the type is columns.
• site.theme, site.seo, site.locales and user.roles are JSON text.
• audit_log gets one row per write, inside the same transaction as the write, with before and after JSON. There is no admin screen for it yet; read it with SQL.

JSON is TEXT on both dialects and the repository layer parses it (parseJson); nothing is stored as jsonb, so an export, a pg_dump and a SQLite file all hold the same strings, and the same toJson/parseJson code runs everywhere. Postgres casts on read where a query reaches into a document ((data::jsonb ->> 'key')). Booleans are INTEGER 0/1 on SQLite and BOOLEAN on Postgres; writes send 0/1 (Postgres coerces) and reads go through toBool().

Two dialects

db/index.ts is the only module that opens a driver. createDb(DATABASE_URL) picks the Kysely dialect from the URL: postgres:// or postgresql:// gives PostgresDialect over a pg.Pool (size DATABASE_POOL_MAX, default 10); sqlite:<path>, a bare path or :memory: gives SqliteDialect over better-sqlite3. The returned handle carries dialect, and dialectOf(db) answers the same for a transaction.

Everything else speaks portable Kysely. Where the two engines disagree, code goes through sqlHelpers(db):

upsert(db, table, conflictColumns, values, update) wraps INSERT ... ON CONFLICT (...) DO UPDATE SET, which both engines support (with excluded.<col> in the update), for counters and rollup tables. likeEscape() escapes a user string for the helpers' LIKE.

Users of the helpers today: collection filters (marketing/collections.ts: JSON field extraction, null-safe neq, case-insensitive contains, the empty-in literal), the entity and media list searches, and the redirect check in review/checks.ts. Migrations call sqlHelpers(db).types.boolean for boolean columns and branch on dialect only for the search table.

Site search is the one feature with two implementations behind one interface (search/provider.ts): search/sqlite.ts uses the FTS5 virtual table with bm25 ranking and snippet() (or a LIKE scan on a build without FTS5); search/postgres.ts queries a generated, weighted tsvector column (title A, body B) with a GIN index, websearch_to_tsquery for all but the last token and a to_tsquery prefix for the last, ts_rank_cd for scores and ts_headline for snippets. marketing/search.ts owns text extraction and index maintenance, which is plain inserts and deletes on both.

Known differences, all at the edges: Postgres applies English stemming and stop words to search (so a prefix on a stemmed form may not match); sorting a collection by a user field whose values mix numbers and text orders them slightly differently; SQLite's lower() is ASCII-only while ILIKE is Unicode-aware.

Tests run on in-memory SQLite by default. With TEST_DATABASE_URL=postgres://... the same suite runs on Postgres: test/helpers.ts creates a test_<random> schema per app, points the pool's search_path at it and drops it on close. CI runs both lanes.

Request flow for a public page

GET /about-us:

1. Security headers, session lookup (anonymous is fine), setup redirect check, then the catch-all in render/routes.ts.
2. The path is decoded and normalized. Paths that start with /_ or /., contain a NUL byte, or end in a file extension (/favicon.ico, /robots.txt) are answered with the 404 page without touching content.
3. Cache lookup by path. On a hit, the stored HTML is returned with X-Spillway-Cache: HIT.
4. On a miss: split a leading locale prefix off the path (/es/about is es + /about; an unprefixed path is the default locale, see localization.md), then find the entity by (site_id, locale, path) (or the home page, or its translation in that locale, for /). If it is not published or has no published revision, 404. Otherwise load the published revision and render from its snapshot, never from the entity's current data, so a draft edit cannot leak.
5. render/page.tsx loads the block library (@spillway-cms/blocks, once per process; a three-block fallback is used if the package cannot be loaded), picks the template (the Puck Render from @puckeditor/core/rsc for types with a layout field, an article template for the rest), computes the <title> (SEO title or title, plus the site suffix), description, canonical URL and Open Graph tags, and renderToStrings the HTML shell with the theme CSS variables and the block CSS inlined in <head>.
6. The HTML is stored in the cache with its tags and returned with X-Spillway-Cache: MISS.

/ with no home page renders a welcome page. /_preview/:id renders the latest revision of any status for a signed-in user with content.view_drafts, with Cache-Control: no-store, X-Spillway-Cache: BYPASS, X-Robots-Tag: noindex and no canonical link.

Cache tags and headers

Every rendered page carries tags: entity:<id>, type:<type key>, site:<site id>. Response headers on public pages:

Cache-Tag: entity:0192...,type:page,site:default
X-Spillway-Cache: HIT | MISS
Cache-Control: public, max-age=0, s-maxage=60, stale-while-revalidate=300

max-age=0 keeps browsers honest; s-maxage and stale-while-revalidate let a CDN serve the page for a minute and refresh in the background (a site sets its own values under Site settings → Cache). Cache-Tag is the header Cloudflare and others use for tag purging; Surrogate-Key carries the same tags for Fastly; Vary stays Accept-Encoding. The CDN purge hook (cdn/) wraps the cache's invalidateTags so every eviction also reaches the site's CDN, coalesced and off the request: caching.md.

The cache (render/cache.ts) is a CacheProvider: get, set (with tags and an optional expiry), invalidateTags, clear, stats. Two providers exist. The default is an in-process LRU map keyed by path. With REDIS_URL set, RedisCacheProvider stores entries in Redis (<prefix>page:<key> with a TTL, <prefix>tag:<tag> sets of keys), keeps a small local LRU in front, invalidates through one pipeline (SMEMBERS of the tag sets, DEL of the pages and sets) and publishes the tags on <prefix>invalidate, so every instance drops the same entries from its local layer. Pages that read the query string (exposed filters, calendars, term listings, the search page) are cached per path?query for a minute, and bounded: a resolver declares the parameters it reads (markQueryDependent(['q', 'level'])), the route keeps only those plus page in the key and in the render (a marker entry at the bare path remembers the list), at most 50 variants per path are kept (least recently used first; a sorted set per path in Redis), and query-dependent entries may take at most 40% of the cache. Invalidation is by tag:

• publishing, unpublishing or deleting an entity evicts entity:<id>, type:<type>, site:<id>, which in practice is every cached page of that site (all its pages share the site tag). This is deliberate: correctness first, the cache refills on the next request;
• editing or restoring a published entity does the same;
• saving a content type evicts type:<key>; saving site settings evicts site:<id>;
• setup clears everything.

Features add tags of their own and evict them from their own writes: collection:<id> and list:<type> (collection blocks and listings), menu:<key>, form:<id>, shared:<key>, media:<id>, term:<id>, tgroup:<id> (translation groups), audience:<key>, experiment:<key>, style:<block>:<key> (block styles), brand-guide (the public style guide), theme, search and registry. The page a feature documents says which tags it uses.

Every public page and every /api/public/render answer also carries a weak ETag (render/etag.ts): a hash of the cached body, kept on the cache entry so a hit never hashes anything. A request whose If-None-Match names it gets a 304 with the cache headers and no body; the 304 carries no Content-Security-Policy, so the browser keeps the policy that matches the nonce in the body it stored. The tag is weak because every response of one entry differs by its nonce.

/sitemap.xml and the per-locale sitemaps are built from the published pages (without reading their data column) and kept in the page cache under site:<id> for up to five minutes (routes/seo.ts), so every publish, unpublish, path move or settings save rebuilds them on the next request.

Responses are compressed through Hono's compress middleware for clients that send Accept-Encoding: gzip (or deflate): HTML, JSON, XML, text, SVG and scripts of 1 KB or more (COMPRESS_THRESHOLD in app.ts); images and other media, HEAD requests, 304s and anything already encoded are sent as they are, and Vary: Accept-Encoding is added. A CDN or reverse proxy that compresses itself may send Accept-Encoding: identity to the origin.

Media files are served with Cache-Control: public, max-age=31536000, immutable; the URL includes the id and filename, so a replaced file gets a new URL. Hashed admin assets get the same; index.html gets no-cache. Image derivatives are generated once per variant (concurrent requests for the same file share one generation) and at most one per core at a time across variants (MAX_CONCURRENT_GENERATIONS in media/derivatives.ts); the rest queue in order.

Auth and CSRF

• Passwords are hashed with scrypt (N=2^15, r=8, p=1, 16-byte salt) and stored as scrypt2$<salt>$<hash> so the algorithm can change later; hashes from the earlier parameters (scrypt$, N=2^14) still verify and are upgraded on the next login.
• Login compares against a dummy hash when the email is unknown, so timing does not reveal valid emails, and is rate limited per client IP (the socket address, or X-Forwarded-For from a proxy named in TRUSTED_PROXIES).
• A session is a 32-byte random token in an httpOnly, SameSite=Lax cookie (spillway_session; __Host-spillway_session with Secure when the site is https). The database stores only an HMAC of the token keyed by SESSION_SECRET, so a copied database cannot forge cookies. Sessions last 30 days and slide, with the row rewritten at most once a day.
• CSRF: every mutating /api request must carry X-Requested-With: spillway. A browser cannot add a custom header cross-origin without a CORS preflight, and Spillway sends no CORS headers, so a form post from another site is rejected with 403 csrf. The admin's API client adds the header to every request.
• Permissions are computed from roles in @spillway-cms/shared (ROLE_PERMISSIONS) and checked in each route with requireUser, requirePermission and requireEditAccess (authors may only touch their own entities).
• Theme values from the admin are stripped of <>{}; before being inlined as CSS. Rich text, the Text block's HTML mode and the Embed block go through the allow-list sanitizer (richtext/sanitize.ts) on save and again at render; see security.md.
• SVG uploads are served with Content-Security-Policy: sandbox so embedded script cannot run in the site's origin. All uploads are sniffed; the stored MIME type comes from the bytes, not the file name.

Multisite

One instance hosts many sites (docs/multisite.md). The site table holds one row per site: the identity (key, domains JSON, primary_domain, status) and the settings (theme, SEO, locales, ...) together. Every per-site table carries site_id (migration 0018_multisite: entity, revision, media, menu, redirect, path_miss, collection, form, form_submission, site_alert, audience, experiment, the analytics rollups and search_index), backfilled with default; keys that used to be unique instance-wide are unique per (site_id, key). Content types, vocabularies, shared blocks, users, tokens, SSO providers, notification rules and modules stay instance-wide.

auth/site.ts runs after the session and bearer middleware: it lists the sites, matches the Host header (lowercased, port stripped) against primary domains and aliases, honors ?__site= or X-Spillway-Site for a user who can open the named site, falls back to the default site, and sets c.var.site and c.var.access. It then recomputes c.var.permissions for that site: global roles ∪ site roles (user_site_role), intersected with a bearer token's subset, plus the ambient part of any section roles. Routes read the site with currentSite(c) and pass its id to the repositories; the renderer keys its cache as <siteId>:<path> and tags pages site:<id>; the redirect map is kept per site; the search providers filter on site_id.

Department delegation: section rows (site_id, key, path_prefix, menu_key) own the entities under a path prefix (entity.section_id, assigned on every save by content/sections.ts, longest prefix first) and user_section_role grants a role inside one section. The guards in auth/guards.ts take the entity (requireEditAccess(c, entity), requireEntityPermission, requireViewAccess) and accept a site-wide permission or a section one; listScope narrows lists to the user's sections. A section-only creator is kept inside the prefix (allowedSectionIds on createEntity/updateEntity).

Optional providers and extension points

• Redis (RedisCacheProvider in render/cache.ts, REDIS_URL): a shared page cache across server processes. Sessions live in the database already; the login and form rate limiters (auth/ratelimit.ts) are per process, so behind a load balancer each process counts its own share.
• Meilisearch (search/meilisearch.ts, SEARCH_URL): SearchProvider (search/provider.ts) has three implementations, SQLite FTS5/LIKE, Postgres tsvector and Meilisearch. All three read the documents marketing/search.ts extracts (title, text and rich text with HTML stripped, layout text, term labels); the SQL ones query the search_index table, Meilisearch receives the same documents on every write and answers with typo tolerance and facets. The listing keyword filter queries search_index too.
• Modules (modules/): a trusted module registers blocks, field types, admin pages, providers, AI tools and hooks on the entity service (entity.beforeSave, afterSave, afterPublish, form.afterSubmit, page.beforeRender) and runs in the process after a graceful restart; a sandboxed extension handles the same hooks as plain Wasm in a worker with declared capabilities and limits. Both come from a signed registry index or an uploaded archive. See modules.md.
• AI (ai/): FieldDef.aiHint, ContentType.description, the brand profile and blockMeta in @spillway-cms/blocks are the prompts' raw material. AI tasks write drafts through the same entity service and permissions as a user and never publish; the MCP server (mcp/) exposes the same operations to agents under an API token.
• CDN purge (cdn/, Site settings → Cache): withCdnHook wraps the page cache so every invalidateTags lands in a scheduler that coalesces tags for two seconds, resolves them per site into tags and public URLs (cdn/urls.ts) and queues one cdn_purge row per site, delivered by deliverCdnPurges with retries on the ticker and audited as cdn.purge. Providers in cdn/cloudflare.ts, fastly.ts, bunny.ts and generic.ts implement CdnPurger; cdn/headers.ts adds Surrogate-Key, Vary and the site's Cache-Control to tagged responses and tags the registry JSON. See caching.md.

Spillway 0.1.0 · Edit this page on GitHub