Benchmarks

How fast one Spillway process is on a realistic site, measured the same way every time, with the numbers from before and after the performance pass and the targets the project aims for. The script is scripts/bench.mjs (pnpm bench); the numbers in benchmarks.json next to this file are the recorded reference for this machine; CI compares a pull request against its base branch measured on the same runner.

What is measured

pnpm bench starts the built server in-process (packages/core/dist) on a temporary SQLite file, and on Postgres as well when TEST_DATABASE_URL is set (a bench_<random> schema, dropped afterwards), and seeds a site through the API:

• the higher-ed kit with its sample content;
• 2,000 more published entities over three types (700 pages built from blocks with rich text in HTML mode and library images, 700 news items with a rich text body and an image, 600 programs with two rich text fields and taxonomy terms), each with 2 to 4 KB of rich text that links other entities and embeds a library image;
• 50 listing pages whose Collection block exposes a keyword search, taxonomy filters and a pager over 10 collections;
• 5 extra locales, with the first 200 news items translated and published in each (1,000 translations);
• 300 redirects, a 20-item main menu linking entities, 200 draft pages for the save scenario, 12 library images of 1200 by 675 pixels.

It then warms the page cache and runs each scenario for 20 seconds (--quick: 5 seconds) with a small load generator built on node:http: 32 keep-alive connections, each sending the next request as soon as the previous answer has been read, Accept-Encoding: gzip, br like a browser. Latency is measured from the start of a request to the end of its body; the first second of each scenario is a warm-up that is not measured. The process's resident set size is sampled every 100 ms.

--only a,b runs a subset, --profile DIR writes a .cpuprofile per scenario (open it in Chrome DevTools or node --prof-process-style tooling), --keep leaves the seeded database in place for EXPLAIN work, --write updates benchmarks.json and the "Latest run" section below, --compare docs/dev/benchmarks.json fails the run when a scenario's p50 is more than 50% (and more than 1 ms) above the baseline; the p95s are printed next to it. The import scenario compares its total time. The median is the metric because a 5 second run holds too few requests for a stable tail: the first second of every scenario is a warm-up whose requests are served but not measured (--warmup), so the queue that forms while 32 connections start at once does not decide the numbers. CI builds the pull request's base branch in a worktree, runs this script against it with --root ../base, then runs the head with --compare against that result, so the gate never depends on the machine the checked-in numbers were recorded on (hosted runners are about twice as slow as the machine behind benchmarks.json). --ignore a,b leaves scenarios out of the gate; by default derivative_first (too few requests in a quick run) and import_crawl (rate-limit bound) are reported but not gated. Refresh benchmarks.json with pnpm bench --write when a change moves the numbers on purpose.

Targets

Stated targets, not promises; the "After" tables say where the build stands on the benchmark machine.

Before

<!-- bench:before:start --> The build before this pass (the same commit's dist before the changes below), run on 2026-11-06: Intel(R) Xeon(R) Processor @ 2.10GHz, 4 cores, 16 GB, Linux 6.18.44-fc-v242, Node v22.23.4; 32 connections, 20 s per scenario.

SQLite

RSS: 142 MB at start, 211 MB after seeding, 265 MB after warming, 986 MB at the end (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.5 s (crawled), 100 ms between requests to the source.

Postgres

RSS: 986 MB at start, 971 MB after seeding, 976 MB after warming, 1379 MB at the end (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.5 s (crawled), 100 ms between requests to the source.

The 50 and 68 errors on api_render_hit are the hit check failing: the scenarios before it had cycled more pages through the cache than it holds, so some of the 50 warm paths had been evicted (the script now warms them again right before the hit scenarios). The 3 errors on collection_filters are requests that took longer than the load generator's connection lived. <!-- bench:before:end -->

After

<!-- bench:after:start --> This build, run on 2026-11-06 at commit 134b88e (its dist also carries the CDN purge hook, the registry organizations and the module blocks that landed on the branch in the meantime; the "Before" build predates them): Intel(R) Xeon(R) Processor @ 2.10GHz, 4 cores, 16 GB, Linux 6.18.44-fc-v242, Node v22.23.4; 32 connections, 20 s per scenario.

SQLite

RSS: 135 MB at start, 210 MB after seeding, 222 MB after warming, 544 MB at the end (after a GC), 552 MB after a second 5 s pass of cache misses (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.5 s (crawled), 100 ms between requests to the source.

Postgres

RSS: 552 MB at start, 556 MB after seeding, 557 MB after warming, 786 MB at the end (after a GC), 787 MB after a second 5 s pass of cache misses (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.6 s (crawled), 100 ms between requests to the source.

How the targets stand on this machine (SQLite): a cache hit takes 0.45 ms at p50 and 0.78 ms at p95 on one connection with gzip, served from the copy compressed once per cache entry (it took 2.2 ms when every hit was compressed on the fly), and compression cuts a 60 KB page to about 12 KB on the wire; a cache miss is 101 ms at p50 and 129 ms at p95 under 32 connections (190 and 284 before); the admin list takes 6.6 ms at p50 and 18 ms at p95 on one connection (admin_list_c1), which under 32 connections queues to 95 to 176 ms at p50 (925 to 1,275 ms before); the 300-page import takes 30.5 s, which is the 100 ms rate limit times 300 pages plus the robots and sitemap reads.

Two numbers moved the wrong way and both are explained by what the response carries, not by the render: page_hit and api_render_hit under 32 connections pay for gzip (see page_hit_identity), and entity_save on SQLite went from 368 to 445 ms at p50: the save's profile shows 60% of it in the search index row (an FTS5 delete that scans the table, unchanged) and 10% in the CDN purge hook's key index walk, which the "Before" build did not have yet. Both were fixed after this run (see "What changed": the search index rowid map and the hook's reverse map); pnpm bench --only entity_save --quick on the fixed build gives 180 ms at p50 on SQLite against 416 ms for the same quick run before it. derivative_first has the throughput of a CPU-bound encoder (AVIF dominates; 6 to 12 requests a second either way) with 400 MB less memory.

The "Before" run measured every request from the first; this run leaves the first second of each scenario out (the warm-up the script now applies), which lowers the tails of the slow scenarios (collection_filters, search) by the queue that forms while 32 connections start at once and changes nothing else by more than noise. <!-- bench:after:end -->

What changed

<!-- bench:changes:start --> Everything below keeps the rendered output, the API responses and the data identical; the profiles (pnpm bench --profile DIR, one .cpuprofile per scenario) and EXPLAIN on both dialects decided what to change.

Looked at and left alone, with the numbers:

• Rich text sanitising at render (richtext/sanitize.ts, content/embed.ts): 2% of a cache miss. Caching sanitised HTML by revision would not measurably help.
• listSites on every request (auth/site.ts): 5% of a cache hit (one row, seven JSON columns). A memo would need cross-instance invalidation for Postgres and Redis deployments; not worth a behavior change for 0.05 ms.
• The save path (entity_save): the search index row and the CDN hook's index walk, the two items the first pass deferred, are fixed above (migration 0033_search_doc and the hook's reverse map). What remains of a save is the revision, the references and the audit row, each one write, and the FTS5 insert itself.
• The admin list returns 50 full entities with their layouts (the contract): JSON.stringify of the response is 23% and parsing the rows 12% of the request. Only a narrower contract would change that.
• Search ranking: FTS5 and ts_rank_cd rank every matching document; the benchmark corpus makes nearly every document match, which is the worst case. Results pages are cached per query for 60 s.
• Body limits (app.ts bodyLimits()) are in place for every route: JSON 1 MB, media 25 MB, module archives 64 MB, registry publishes 20 MB, the form submit and the beacon 256 KB. <!-- bench:changes:end -->

Memory

<!-- bench:memory:start --> The resident set after the whole SQLite run and a forced GC is 544 MB, and a second five-second pass of cache misses over every page moves it to 552 MB: the caches are full and bounded, nothing accumulates per request. The "Before" build ended the same run at 986 MB (the sitemap scenario alone added 450 MB of parsed data columns, and 32 concurrent sharp pipelines another 300 MB). What holds memory in a loaded process, and its bound:

The gap between 222 MB after warming and 544 MB at the end is the full page cache (500 pages of 60 to 100 KB plus their JSON twins), libvips's buffers and V8 heap that a GC does not hand back to the OS; it does not grow with more requests. <!-- bench:memory:end -->

Latest run

The section below is rewritten by pnpm bench --write.

<!-- bench:latest:start --> Run on 2026-11-06 at commit 134b88e: Intel(R) Xeon(R) Processor @ 2.10GHz, 4 cores, 16 GB, Linux 6.18.44-fc-v242, Node v22.23.4; 32 connections, 20 s per scenario.

SQLite

RSS: 135 MB at start, 210 MB after seeding, 222 MB after warming, 544 MB at the end (after a GC), 552 MB after a second 5 s pass of cache misses (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.5 s (crawled), 100 ms between requests to the source.

Postgres

RSS: 552 MB at start, 556 MB after seeding, 557 MB after warming, 786 MB at the end (after a GC), 787 MB after a second 5 s pass of cache misses (after a GC); page cache 500/500 entries. Import crawl: 300 pages in 30.6 s (crawled), 100 ms between requests to the source. <!-- bench:latest:end -->

Spillway 0.1.0 · Edit this page on GitHub