Fleet: provisioning and managing many Spillway sites
@spillway-cms/fleet (packages/fleet) creates and operates Spillway sites on a host from one command or one HTTP call. It is what a hosting provider such as Accorbis runs to turn "new customer" into "running site"; anyone can use the same CLI for their own sites.
One call does all of this: create the app and its volume (or point it at Postgres), set the secrets, start a machine from the ghcr.io/kwall-llc/spillway-cms image, wait for /healthz, run the setup wizard (POST /api/setup) with the site name and the first admin, apply a content template, attach the customer domain with a certificate, and record the site in the fleet registry. Target: under five minutes, most of it the image pull.
spillway-fleet provision --name "Acme Inc" --admin-email owner@acme.test --admin-name Owner \
--template marketing --domain www.acme.test
The Accorbis flow
Accorbis spillway-fleet Fly.io (Machines API + GraphQL) the new site
| | | |
| POST /sites {name, admin, | | |
| template, domain, profile} | | |
|------------------------------->| 1 registry: site_xxx "creating" | |
| | event site.provisioning --> webhook |
| | 2 POST /v1/apps {app_name, org} | |
| | allocateIpAddress shared_v4 + v6 | |
| | setSecrets SESSION_SECRET [, DATABASE_URL] |
| | POST /v1/apps/{app}/volumes 1 GB | |
| | POST /v1/apps/{app}/machines {image, env, mounts /data, services} |
| | GET .../machines/{id}/wait?state=started |
| | event site.created | |
| | 3 GET https://{app}.fly.dev/healthz every 2 s until 200 ----------------->|
| | 4 POST /api/setup {siteName, email, password, name} --------------------->|
| | 5 POST /api/types, POST /api/entities, POST .../publish, PUT /api/site --->|
| | 6 addCertificate(app, domain) ----->| |
| | event site.domain_added | |
| | 7 registry "ready"; event site.ready |
| 201 SiteRecord {id, url, | | |
| adminUrl, adminPassword, dns}| | |
|<-------------------------------| | |
Every step is idempotent. If anything fails, the registry keeps the record in state failed with the step and the error; calling provision again with the same name resumes: an app, volume or machine that already exists is reused, /api/setup is skipped when the site reports installed: true, and the template is only applied on the run that did the setup.
Install and run
The package builds with the rest of the workspace (pnpm build) and exposes the spillway-fleet binary (packages/fleet/dist/cli.js). From a checkout:
pnpm build
alias spillway-fleet='node packages/fleet/dist/cli.js'
export FLY_API_TOKEN=... FLY_ORG=my-org FLY_REGION=ams
spillway-fleet provision --name "Acme Inc" --admin-email owner@acme.test --admin-name Owner --template marketing
The result prints the URL, the admin URL, the admin email and the generated password (only when you did not pass --admin-password; it is never written to disk). Use --json on any command for machine-readable output.
Environment variables
| Variable | Default | What it does |
|---|---|---|
FLEET_HOST | fly | Default --host: fly, docker, kubernetes, railway, render. |
FLEET_STATE | ~/.spillway/fleet.json | The registry file (mode 0600). |
FLEET_SECRET_KEY | unset | 32 bytes (base64url, or 64 hex characters; openssl rand -base64 32) that seal the site tokens in the registry. Unset: fleet.key next to the registry file, generated on first use with mode 0600. Keep it with the registry: without it the sealed tokens cannot be opened (renew them with plan token). |
FLEET_WEBHOOK_URL | unset | Receives every lifecycle event as a JSON POST (see below). |
FLEET_WEBHOOK_SECRET | unset | When set, each webhook body is signed: X-Spillway-Signature: sha256=<hex HMAC-SHA256 of the body>. |
FLEET_IMAGE | ghcr.io/kwall-llc/spillway-cms | Base image; --tag is appended (the release workflow publishes vX.Y.Z, X.Y and latest; see dev/releases.md). A full reference with / or @ as the tag is used verbatim. |
FLEET_TOKEN | unset | Bearer token for serve (or --token). At least 16 characters. |
FLY_API_TOKEN | required for fly | Token for api.machines.dev and api.fly.io/graphql (fly tokens create org). |
FLY_ORG | required for fly | Organization slug the apps are created in. |
FLY_REGION | ams | Region for the volume and the machine; --region overrides per site. |
FLY_MPG | unset | 1 asks for a Fly Managed Postgres cluster on the standard profile. Not supported yet: the call fails with not_supported and tells you to create the cluster yourself. |
FLY_SECRETS_MODE | graphql | graphql sets app secrets through setSecrets; env writes them into the machine config instead (visible in the API; only for debugging). |
FLY_MIN_MACHINES | 0 | min_machines_running in the machine's service. 0 lets the machine stop when idle. A plan sets its own (see Plans). |
FLY_METRICS_TOKEN | FLY_API_TOKEN | Read token for Fly's Prometheus API (fly tokens create readonly), used by auto run. |
FLY_PROMETHEUS_URL | https://api.fly.io/prometheus/<FLY_ORG> | Override for the metrics endpoint (the tests point it at a fake). |
FLY_API_BASE, FLY_GRAPHQL_URL | the public endpoints | Overrides, used by the tests to point at a fake. |
DOCKER_HOST | unix:///var/run/docker.sock | Docker Engine endpoint; tcp://host:2375 or http://host:2375 for a remote daemon (no TLS; use an SSH tunnel). |
FLEET_DOCKER_PORT | Docker picks one | Host port to publish 3000 on. |
FLEET_DOCKER_PUBLIC_HOST | 127.0.0.1 | Host name used in the site URL and in the DNS hint. |
FLEET_K8S_NAMESPACE | spillway | Namespace for the rendered manifests. |
FLEET_K8S_DIR | ~/.spillway/k8s | Where manifests and spec.json are written, one folder per site. |
FLEET_K8S_INGRESS_CLASS, FLEET_K8S_STORAGE_CLASS, FLEET_K8S_CLUSTER_ISSUER | unset | Ingress class, PVC storage class, cert-manager ClusterIssuer annotation. |
Provisioning also sets these on the site's container: PORT=3000, DATA_DIR=/data, BASE_URL (https://<domain> when a domain is given, else the host's URL), SESSION_SECRET (generated, 48 random bytes), SETUP_TOKEN (generated, 32 random bytes; handed to the host as a secret like SESSION_SECRET) and DATABASE_URL on the standard profile. --env KEY=VALUE adds more.
SETUP_TOKEN closes the window between the site answering on its public address and the fleet running the setup wizard: the site refuses POST /api/setup unless the request carries the token in X-Setup-Token, and only the provisioning process knows it (the fleet sends it with its own setup call). A site that reports installed: true before the fleet ran setup is still recorded with setupSkipped: true; with the token in place that can only be a retry of an earlier provisioning, not a stranger's wizard run. domain values are validated as host names (RFC 1123 labels under a real TLD) on POST /sites, POST /sites/:id/domains and in the CLI before they reach a Traefik rule, an Ingress or BASE_URL.
Profiles
single(default): one machine, SQLite on a 1 GB volume mounted at/data. The right choice for a marketing site.standard: Postgres throughDATABASE_URL(--database-url), for more than one instance or a managed database. The/datavolume is still created, because uploads and the generated secrets live underDATA_DIReven with Postgres (see install.md). On Fly, create the database first (fly postgres create, then take the URLfly postgres attachprints, or any Postgres reachable from the app's network).FLY_MPG=1is reserved for Fly Managed Postgres and answers "not supported yet" until that API is verified.
Plans
A plan turns "what the customer bought" into machine sizes, volumes, a Postgres tier, add-ons and CDN headers. The catalogue is data in packages/fleet/src/plans.ts (PLANS, planFor, resolvePlan); the prices, what each plan includes and what it costs on Fly are in dev/hosting-plans.md. Plans need the Fly host for everything past the first machine; other adapters receive the plan's machine size and ignore the rest.
| Plan | Price | Runs on |
|---|---|---|
site | $49 per site | one shared-1x 512 MB machine that sleeps when idle, SQLite on a 10 GB volume, CDN headers 300 / 3600 |
campus | $249 per site | one shared-2x 2 GB machine always on, Managed Postgres Basic with HA (or --database sqlite), a staging app that sleeps, media on R2 (or --storage tigris), CDN headers 600 / 3600 |
enterprise | $1,499 per organization (10 sites) | two performance-2x 8 GB machines always on, each with its own volume, Managed Postgres Launch with HA, Upstash Redis, Meilisearch, a staging app, media on R2 (or --storage tigris), CDN headers 3600 / 86400 |
auto | the customer's monthly maximum (at least $49) | the richest of the three whose cost at its limits fits in 70% of the maximum, kept under it by the Auto controller |
spillway-fleet provision --name "Acme Inc" --admin-email owner@acme.test --admin-name Owner \
--plan campus --database-url postgres://... # or --database sqlite
spillway-fleet provision --name "Small Co" --admin-email o@small.test --admin-name Owner --plan auto --max-monthly 80
spillway-fleet provision --name "Fly Only" --admin-email o@fly.test --admin-name Owner --plan campus --database sqlite --storage tigris
What provisioning on a plan does, on top of the steps above:
- Machines. The web machine gets the plan's size (
guest),autostop: "stop"withmin_machines_running: 0on Site (it sleeps),autostop: "off"withmin_machines_runningequal to the replica count on Campus and Enterprise. Enterprise gets a second web machine (<app>-web-2) with its own volume (spillway_data_2); Fly volumes attach to one machine each. Machines carry the metadataspillway.plan. - Database. Campus and Enterprise run on Managed Postgres, which the fleet does not create (the MPG API is not public): without
--database-urlthe request is refused with thefly mpg create ... --plan <tier>command to run, and on Campus the hint to choose--database sqliteinstead. The plan's tier and HA are recorded on the site record. - Add-ons. Enterprise gets Meilisearch as a companion app
<app>-search(one shared-1x 1 GB machine with a 10 GB volume, reached on the private network:SEARCH_URLas plain env,SEARCH_API_KEYas a secret) and Upstash Redis through Fly's add-on API (<app>-cache, the fixed 1 GB plan;REDIS_URLas a secret). - Staging. Campus and Enterprise get a companion app
<app>-stagingwith its own small machine and volume that sleeps when idle, its ownSESSION_SECRET, set up with the same admin login as production and with CDN headers 0 / 0. - The site token. Right after setup, from the admin's session, the fleet mints an API token on the site (
POST /api/tokens, labelspillway-fleet <site id>, permissionsite.settingsonly, no expiry) and keeps it on the site record, sealed with the registry key (AES-256-GCM, never in clear; see Registry and webhook). Every later call the fleet makes to the site settings API uses it asAuthorization: Bearer spw_.... - CDN headers. The plan's
s-maxageandstale-while-revalidatego into Site settings → Cache throughPUT /api/sitewith the token (the CDN provider and its credentials stay as they are). How these headers behave with and without a purge provider is in caching.md. - Object storage. Campus and Enterprise put media on object storage behind the CDN: Cloudflare R2 by default, Tigris with
--storage tigrisfor a customer who wants everything on the Fly invoice. The choice is recorded on the plan (plan.media.objectStorage), the cost model prices it (dev/hosting-plans.md), and a plan change keeps it unless--storagesays otherwise. Until the core stores uploads there, the fleet records the choice and the uploads stay on the volume. - The record. The resolved plan (
plan), the companion apps and add-ons (details.plan) and, on Auto, the controller state (auto) are written to the site record.
A provisioning that resumes on a site that is already set up has no session to mint a token with; the result says so, and spillway-fleet plan token <id> --admin-password <password> logs in, mints one, stores it and applies the plan's CDN headers. The same command replaces a token that was revoked in the admin (it revokes the previous one).
Showing and changing a plan
spillway-fleet plan list prints the catalogue with each plan's Fly cost at its limits for --region (default FLY_REGION). spillway-fleet plan show <id> prints the site's plan, its cost at the plan's limits line by line, the site token's prefix and, on Auto, the cap, the applied steps, the last spend and projection, the warnings and the last audit lines.
spillway-fleet plan change <id> --plan enterprise [--storage r2|tigris] moves a running site, adding before removing:
- companions and add-ons the new plan has are created, or started again when an earlier change or the Auto controller stopped them (Meilisearch, Redis, staging; a new staging app is set up with a generated admin password that the command prints once);
- new replicas are created and started;
- the machines that stay are updated one at a time (size, autostop, environment), each waited for;
- surplus replicas are destroyed, last, with their volumes kept;
- companions the new plan drops are stopped, never deleted, and listed under
details.plan.retired(a later plan that has them starts them again); a detached Redis add-on keeps billing until you delete it (fly redis destroy <name>), and the command says so; - the new plan's CDN headers go through the site settings API, the record takes the new plan, and
site.plan_changedis emitted.
Limits: a web machine carries its /data volume, which attaches to one machine, so a machine that stays is updated in place (Fly restarts it with the new size, a few seconds); with two replicas the other one serves meanwhile, and with one the restart is the only interruption. A plan change never moves the database between SQLite and Postgres (that is spillway export, a new site on the new plan, spillway import), never shrinks or grows a volume (fly volumes extend), and does not change the Managed Postgres tier (the command notes the tier the new plan expects). Moving onto Auto, or changing the Auto maximum, clears the applied steps and starts the ladder again from the new stack.
Auto
On the Auto plan the customer sets a monthly maximum: at least $49, a hard cap. The fleet keeps the site inside it with a controller that degrades the site in a published order rather than billing more or going dark, and restores it when the month allows. Three rules hold throughout: never delete data, never stop serving, never bill above the cap (the cap is what the customer is billed at most; spend the ladder cannot avoid is the platform's). No step deletes a volume, an app, a database or object storage; the one thing the controller deletes is the Upstash Redis add-on, which holds only a page cache the site rebuilds, and it is recreated on restore.
spillway-fleet auto run --interval 5m # every 5 minutes until stopped (SIGINT or SIGTERM)
spillway-fleet auto run --once # one pass, for a scheduler
spillway-fleet auto run --dry-run [--site id] # what the next pass would do, changing nothing
Each pass looks at every ready Fly site on the Auto plan (packages/fleet/src/auto.ts, autoTick):
- It reads this month's usage from Fly's Prometheus API (
https://api.fly.io/prometheus/<org>/api/v1/query): machine seconds per app (the site, its staging and its Meilisearch app), egress at the edge, volume fill. - It prices the spend to date and projects the month end at Fly's rates (dev/hosting-plans.md).
- Up: when the projection exceeds the cap, it applies the next step of the ladder, one step per pass, and only once the step before has been in place for its dwell time.
- Down: when the projection has stayed under 70% of the cap for 24 hours, it restores the last applied step, again after that step's dwell; the next restore waits another 24 hours.
- Hold: between 70% and 100% of the cap nothing moves, and the 24 hours start again (hysteresis: a month near the cap does not flap).
- Follow-ups: while the
redisstep is applied and its add-on still exists, the pass checks the site's/healthz; once the site has answered at every pass for the step's dwell time on its in-process cache, it deletes the Upstash Redis add-on (below). A failed check starts the dwell again. - It writes one audit line per action on the site record (the last 200: time, action, step, spend to date, projection, cap, what was done), replaces the record's warnings, and emits
site.auto_stepwhen a step was applied or restored or the Redis add-on was deleted (data.actionisdegrade,restoreordelete).
The ladder, in order. A step that does not apply to the site's stack (no second replica, no staging, no Meilisearch, no Redis, an origin that already sleeps) is skipped.
| Tier | Step | Dwell | What it does | What a customer sees |
|---|---|---|---|---|
| 1 | cache | 30 min | Sets the CDN headers through the site settings API to four times the plan's s-maxage (at least 3600) and twice its stale-while-revalidate (at least 86400). Fewer requests reach the origin, so less egress. | Pages are held at the edge longer and served stale while the origin refreshes them. With a purge provider connected a published change still reaches visitors within seconds; without one it can take up to the new s-maxage. |
| 2 | replicas | 60 min | Destroys the second web machine; its volume is kept. | One web machine instead of two: no spare capacity for a spike, and a deploy restarts the site for a few seconds. |
| 2 | downsize | 60 min | Moves the web machines one size down (performance-2x 8 GB to 4 GB, shared-2x 2 GB to shared-1x 1 GB, ...). | Slower renders on cache misses and in the admin; cached pages are as fast as before. |
| 3 | staging | 60 min | Suspends the staging machine (its volume and data stay). | The staging instance does not answer until the step is restored. |
| 3 | search | 60 min | Removes SEARCH_URL from the site and stops the Meilisearch machine (its index stays on its volume). | Site search falls back to the database's full-text search: no typo tolerance or facets, results still complete because the SQL index is kept in sync. |
| 3 | redis | 60 min | Unsets REDIS_URL; the site uses its in-process page cache. Once the site has served cleanly on it for the dwell time (its health check passing at every pass), deletes the Upstash add-on through Fly's API and records the add-on's plan and region on the site record (auto.redis). The cache is disposable, so nothing is lost. | Each web machine warms its own cache; the first request for a page after a change can be slower. After a restore the shared cache starts empty and warms again. |
| 4 | sleep | 120 min | Sets autostop: "stop" and min_machines_running: 0: the origin sleeps between requests and starts on the next one. | The first request after a quiet spell takes a few seconds while the machine starts; the CDN keeps serving cached pages meanwhile. |
Restoring walks the same table bottom up: the origin stays awake again; the Redis add-on is created again with the recorded plan and region (or found, when it was not deleted yet), pinged from the site's web machine, REDIS_URL is set, and only then are the machines restarted on it (the core refuses to boot on a Redis that does not answer, so a failed ping leaves the site on its in-process cache, records the failure and tries again on the next pass); Meilisearch starts and SEARCH_URL comes back (run spillway search reindex to catch up with what was published meanwhile), staging starts, the machines go back to the plan's size, the replica is recreated on its old volume, the plan's CDN headers return.
When every applicable step is applied and the projection is still over the cap, the site keeps serving at that configuration, the warning cap reached is recorded, and the rest of the month is the platform's cost. A volume at 80% or more raises a storage warning: nothing is deleted to make room, media and backups keep growing until someone acts.
--dry-run runs the same reading and decision, prints the spend lines, the projection, the applied steps and the step the next pass would apply or restore with its reason, and changes nothing: no Fly call, no site call, no registry write.
Limits:
- Fly has no budget API. There is no way to tell Fly "stop at $X"; the fleet computes spend itself from metrics at the rates in
pricing.ts. The numbers are as good as those rates and the metrics; re-read the rates before the first invoice, and check the Prometheus label names against a real organization (see What is not verified). - Egress cannot be capped. Bytes a visitor pulls are billed whether or not the controller likes it, and a hot file served from the origin can outrun every step. Media belongs on object storage behind the CDN (Campus and Enterprise plan for it), so the origin serves HTML and the CDN serves files; the
cachestep is the lever that reduces origin egress. - What a step keeps keeps billing. A removed replica's volume and a stopped Meilisearch machine's volume are kept (never delete data) and still cost money; the projection counts them. The Redis add-on is the exception: its contents are a page cache the site rebuilds, so the
redisstep deletes it after its dwell, and from then on the projection leaves out its $20 a month. Until the deletion it bills and is counted. - Warnings. While the add-on is deleted the record carries
redis: Redis add-on <name> deleted, recreated on restore (plan ..., region ...); a failed health check during the dwell adds a warning that the add-on is kept and the dwell starts again. - One step per pass, after its dwell. A sudden spike late in the month can move the projection past the cap faster than the ladder walks; the dwell times keep the controller from thrashing machines, and the cap rule above covers the gap.
- Fly only. The controller drives the Machines API; Docker and Kubernetes sites cannot be on Auto.
CLI
spillway-fleet provision --name NAME --admin-email E --admin-name N [--host H] [--template T] [--admin-password P]
[--domain D] [--profile single|standard] [--region R] [--tag TAG] [--app-name A]
[--database-url URL] [--env KEY=VALUE ...] [--plan site|campus|enterprise|auto]
[--max-monthly USD] [--database sqlite|postgres] [--storage r2|tigris] [--json]
spillway-fleet plan list [--region R] [--json] the catalogue with each plan's Fly cost at its limits
spillway-fleet plan show <id> [--json] the site's plan, its cost, the Auto state
spillway-fleet plan change <id> --plan P [--max-monthly USD] [--storage r2|tigris] [--json]
spillway-fleet plan token <id> --admin-password P [--admin-email E] mint and store a new site token
spillway-fleet auto run [--interval 5m] [--once] [--dry-run] [--site id,id] [--json]
spillway-fleet list [--json]
spillway-fleet status <id> [--json] registry record + live host status, /healthz and version
spillway-fleet upgrade <id> --tag TAG deploy the image, wait for /healthz
spillway-fleet upgrade-all --tag TAG [--batch 5] [--pause 60] [--only id,id]
spillway-fleet secrets set <id> KEY=VALUE ...
spillway-fleet secrets recovery <id>
spillway-fleet backup <id> runs `spillway export` inside the machine
spillway-fleet destroy <id> --yes
spillway-fleet domain add <id> <domain>
spillway-fleet register --host H --app-name A --url URL add a hand-made site to the registry
spillway-fleet render-config --host railway|render --name NAME [--domain D] [--tag TAG] [--out DIR]
spillway-fleet templates
spillway-fleet serve [--port 4000] --token <bearer>
Backup keys and recovery. Every site the fleet provisions gets its own BACKUP_ENCRYPTION_KEY (64 hex characters) next to SESSION_SECRET, so its backups, which carry the instance's key files, are always encrypted. The fleet keeps both values sealed on the site record (recoverySecrets, AES-256-GCM under the registry key, never in events or the HTTP service). If a site's host app is lost, spillway-fleet secrets recovery <id> prints them so a new instance can restore the site's backups; a key passed with --env BACKUP_ENCRYPTION_KEY=... at provision time is used and kept instead.
<id> is the registry id (site_...), the app name or the domain.
upgrade-all is a staged rollout: sites are taken in creation order, --batch at a time in parallel, with --pause seconds between batches. Each site is deployed and then polled on /healthz; the first failed health check marks that site failed, finishes its batch and stops the rollout (stoppedEarly: true, exit code 1). Sites not in state ready are skipped. Fix the failed site (upgrade <id> --tag <previous> rolls it back) and run upgrade-all again; sites already on the new tag are deployed again, which is a no-op for Fly.
backup runs node /app/packages/core/dist/cli.js export /data/exports/export-<timestamp>.json inside the machine (Fly exec API, Docker exec, kubectl exec) and prints the command to fetch the file (fly ssh sftp get ..., docker cp ..., kubectl cp ...). The export holds settings, types, entities, revisions, users (no hashes) and media metadata; uploaded files stay in /data/media and need their own copy.
HTTP service
spillway-fleet serve --port 4000 --token <bearer> runs the same operations behind bearer auth. This is the endpoint Accorbis calls. Put it behind TLS; the token is compared in constant time and every response is JSON.
| Method and path | Body | Answer |
|---|---|---|
GET /healthz | { ok, sites } (no auth) | |
POST /sites | { host, name, adminEmail, adminName, template?, adminPassword?, domain?, profile?, region?, imageTag?, appName?, databaseUrl?, env?, plan?, maxMonthlyUsd?, database? } | 201 SiteRecord with adminPassword when generated, templateSummary, details.domain.dns. The call waits for the site (minutes); add ?async=1 to get 202 { accepted, site } at once and poll GET /sites/:id. |
GET /sites | SiteRecord[] (never with passwords; siteToken without its sealed secret) | |
GET /sites/:id | SiteRecord; ?live=1 adds { host, healthy, version } | |
POST /sites/:id/upgrade | { tag } | SiteRecord |
POST /sites/:id/domains | { domain } | { domain, dns: [{ type, name, value }], details } |
POST /sites/:id/secrets | { env: { KEY: value } } | { ok, keys } |
POST /sites/:id/backup | { location, howToFetch } | |
DELETE /sites/:id | { destroyed: SiteRecord } |
Errors: { error: { code, message, details? } } with 400 (validation_error), 401 (unauthorized), 404 (not_found), 405, 501 (not_supported), 502 (host_api_error, provision_failed, upgrade_failed, spillway_api_error), 500.
curl -s -X POST http://fleet.internal:4000/sites \
-H "Authorization: Bearer $FLEET_TOKEN" -H 'Content-Type: application/json' \
-d '{"host":"fly","name":"Acme Inc","adminEmail":"owner@acme.test","adminName":"Owner","template":"marketing","domain":"www.acme.test"}'
SiteRecord
{
"id": "site_8f3c2a1b9d4e", "host": "fly", "hostId": "spillway-acme-inc", "appName": "spillway-acme-inc",
"name": "Acme Inc", "url": "https://spillway-acme-inc.fly.dev", "adminUrl": "https://spillway-acme-inc.fly.dev/admin",
"adminEmail": "owner@acme.test", "adminPassword": "only in the provisioning response, only when generated",
"region": "ams", "profile": "single", "image": "ghcr.io/kwall-llc/spillway-cms:latest", "domain": "www.acme.test",
"template": "marketing", "state": "ready", "createdAt": "...", "updatedAt": "...",
"details": { "machineId": "...", "volumeId": "vol_...", "ips": [...], "domain": { "dns": [...] }, "plan": { "machineIds": [...], "staging": {...} } },
"plan": { "name": "campus", "base": "campus", "label": "Campus", "priceUsd": 249, "web": {...}, "database": {...}, "cdn": {...}, ... },
"auto": { "enabled": true, "maxMonthlyUsd": 80, "applied": ["cache"], "warnings": [], "audit": [...] },
"siteToken": { "id": "...", "prefix": "spw_ab12", "label": "spillway-fleet site_8f3c2a1b9d4e", "permissions": ["site.settings"], "createdAt": "..." }
}
plan and siteToken are present on sites provisioned with --plan, auto on the Auto plan only. In the registry file siteToken also holds sealed, the token encrypted with the registry key; the service and the webhook never send it.
States: creating, waiting (for /healthz), setup, ready, upgrading, failed (with error), destroyed.
Registry and webhook
The registry is a JSON file: { "version": 1, "sites": [SiteRecord...] }, written atomically, mode 0600, never containing passwords. The one secret it holds, each site's API token, is sealed: AES-256-GCM under FLEET_SECRET_KEY or the generated fleet.key next to the registry (mode 0600), stored as v1:<iv>:<tag>:<data>, the scheme the core uses for provider keys. A copied registry file alone reveals no token; back up the key with it, or renew the tokens with plan token after a restore. Back it up with the rest of your configuration; spillway-fleet register rebuilds entries for sites that exist on the host, and provision resumes safely when a record is missing (it finds the app on the host and skips setup when the site is installed).
Every lifecycle change is also sent to FLEET_WEBHOOK_URL:
{ "id": "evt_...", "kind": "site.ready", "at": "2026-01-01T00:00:00.000Z", "siteId": "site_...", "site": { ...SiteRecord }, "data": { "setupSkipped": false, "template": "marketing" } }
Kinds: site.provisioning, site.created, site.ready, site.failed (data.step, data.error), site.upgrading, site.upgraded, site.secrets_set (data.keys), site.domain_added (data.dns), site.backup, site.plan_changed (data.from, data.to, data.steps, data.notes), site.auto_step (data.action, data.step, data.spend, data.projected, data.cap, data.applied, data.warnings), site.destroyed. Headers: X-Spillway-Event: <kind> and, with FLEET_WEBHOOK_SECRET, X-Spillway-Signature: sha256=<hmac>. A failed delivery is logged and never fails the operation; the registry file remains the source of truth. Accorbis mirrors these into its own dashboard.
Templates
A template is a Spillway export file (spillway export, format: "spillway-export", version: 1) with two optional fleet extensions: description and settings (merged into the site through PUT /api/site). --template marketing resolves to packages/fleet/templates/marketing.json; anything with a slash or .json is a path. Applying goes through the public API as the new admin: types are created (or updated when the key exists), entities are created and published when the template marks them published, then settings are merged. Entity ids in the file are not kept and references between template entities are not remapped, so keep templates self-contained. The same files load with spillway import (ids, revisions and paths are complete).
Shipped templates, both verified against the core (through importData and through the HTTP flow, every page renders):
marketing: Product, Pricing and Contact pages, a first post,case_study(/customers/{slug}) andtestimonialtypes with one example each,llms.txton.institutional: About, Governance, Departments and Publications pages, a welcome post,department(/departments/{slug}) andpublication(/publications/{slug}, with a reference to department) types with one example each.
Make your own: set a site up the way you want, run spillway export my-template.json, remove users and media (files are not carried), add a description, and pass the path to --template.
Adapters
| Fly | Docker | Kubernetes | Railway | Render | |
|---|---|---|---|---|---|
| createSite | app, IPs, secrets, volume, machine | volume, container, start | manifests + kubectl apply | renders config, 501 | renders config, 501 |
| getStatus | machine state | inspect (running, health) | kubectl get deploy | 501 | 501 |
| setSecrets | setSecrets + machine update | recreate container with merged env | re-render Secret, rollout restart | 501 | 501 |
| deploy | machine update with new image | pull, recreate container | re-render, apply, rollout status | 501 | 501 |
| attachDomain | addCertificate + DNS hints | label + BASE_URL, Traefik labels | Ingress with TLS block | 501 | 501 |
| listSites | apps in the org with prefix spillway- | containers with label spillway.fleet=1 | folders with a spec.json | [] | [] |
| backup | exec spillway export | exec spillway export | kubectl exec | 501 | 501 |
Fly (packages/fleet/src/adapters/fly.ts). Machine defaults: shared-cpu-1x, 512 MB, volume 1 GB encrypted, restart: on-failure, one service on port 3000 with autostop: "stop", autostart: true, min_machines_running: 0, HTTP check on /healthz every 15 s, ports 80 (force HTTPS) and 443. Secrets go through GraphQL setSecrets before the machine is created so they are injected at boot; setSecrets later re-posts each machine's config so the new release reaches it. A shared IPv4 and an IPv6 are allocated, otherwise the app has no public endpoint. attachDomain returns the DNS the customer must set: CNAME <domain> -> <app>.fly.dev for subdomains, A/AAAA with the app's IPs for apex domains, plus the _acme-challenge CNAME when Fly asks for DNS validation. destroy deletes the app and its -staging and -search companions, which takes their machines and volumes with them; it is the only command that deletes a volume. Plans add the companion apps, the Redis add-on, replicas and changePlan (see Plans). Limits: one machine per site on SQLite; listSites can only tell fleet apps apart by the spillway- prefix; no file download API, so backup leaves the export on the volume.
Docker (docker.ts): local development and Coolify-style VPS hosts through the Docker Engine API. The container gets labels (spillway.fleet, spillway.site, spillway.domain, Traefik router labels when a domain is given) and a named volume <app>-data:/data. Changing env or image means recreating the container; the published host port is kept across recreations. TLS and routing are the reverse proxy's job (Coolify's Traefik reads the labels). Not supported: TLS-protected daemons, certificates.
Kubernetes (kubernetes.ts): renders Namespace, Secret (SESSION_SECRET, DATABASE_URL), PersistentVolumeClaim, Deployment (Recreate strategy, readiness and liveness probes on /healthz, fsGroup: 1000), Service and, with a domain, an Ingress with a TLS block and the cert-manager annotation. With kubectl on PATH the manifests are applied and the rollout awaited; without it they are written to FLEET_K8S_DIR/<app>/ and returned in details.manifests. Limits: without a domain the site URL is the in-cluster service address, so provisioning from outside needs a domain and an ingress controller; status and backup need kubectl.
Railway and Render (manual.ts): no API is driven. render-config writes railway.json + a one-line Dockerfile + .env.railway, or render.yaml (image runtime, disk at /data, health check /healthz, generated SESSION_SECRET), and prints the manual steps. Once the service runs, spillway-fleet register adds it to the registry so list/status see it; upgrade, secrets, backup, domain and destroy answer 501 not_supported with the dashboard step to take instead.
Cost notes
- Fly: the machine stops when idle (
autostop: "stop",min_machines_running: 0) and starts on the first request, so a quiet site costs the volume plus the seconds it runs. The first request after idle takes a few seconds while SQLite opens and the page cache warms; setFLY_MIN_MACHINES=1for sites that need it always warm.shared-cpu-1xwith 512 MB is enough for a marketing site;sizein the spec maps toguest. - The volume is 1 GB without a plan and 10 GB on every plan; uploads grow it. Fly volumes are per machine, so a SQLite site is one machine, and each Enterprise replica has its own volume until media moves to object storage.
- Plans and Auto: the cost of each plan at Fly's rates is in dev/hosting-plans.md;
spillway-fleet plan listandplan showprint it. - Docker on a VPS: many sites share one daemon; memory is capped per container (
spec.size.memoryMb, default 512 MB). - Kubernetes: requests
100mCPU /256Mi, limit 512 MB by default. - Registry and webhook cost nothing: a file and a POST.
What is not verified
No real Fly.io, Docker, Kubernetes, Railway or Render call was made while building this package. Every adapter was exercised against in-process fakes (
packages/fleet/src/test/fakes.ts) that answer with bodies shaped from the public API documentation as remembered, so the request and response shapes below need one supervised run against a real account before Accorbis relies on them. The templates, the registry, the webhook, the staged rollout logic and the HTTP service are tested for real; the core's/api/setupand import paths were run against the actual server.
Fly assumptions to confirm (all in the header comment of fly.ts):
- Machines API paths and bodies:
POST /v1/apps {app_name, org_slug},GET/DELETE /v1/apps/{app},GET /v1/apps?org_slug=,POST /v1/apps/{app}/volumes {name, region, size_gb, encrypted},POST /v1/apps/{app}/machines {name, region, config},POST /v1/apps/{app}/machines/{id} {config}for updates,GET .../wait?state=started&timeout=60,POST .../exec {command: string[], timeout}and its{exit_code, stdout, stderr}answer. services[].autostopas the string"stop"(older clients send a boolean).- GraphQL:
setSecrets(input: {appId, secrets: [{key, value}]}),allocateIpAddress(input: {appId, type: shared_v4 | v6}),addCertificate(appId, hostname)and the certificate fieldsdnsValidationHostname/dnsValidationTarget; thatappIdaccepts the app name. - That secrets set through GraphQL are injected into a machine created afterwards without an extra release step, and that re-posting the machine config picks up a later
setSecrets. - That
DELETE /v1/apps/{app}removes machines and volumes. - Managed Postgres: no API is called;
FLY_MPG=1is an explicitnot_supportederror. - Plans:
DELETE /v1/apps/{app}/machines/{id}?force=truekeeping the volume,POST .../machines/{id}/start|stop|suspend(asuspendanswered 405 or 422 falls back tostop), GraphQLunsetSecrets(input: {appId, keys}),createAddOn(input: {organizationId, type: upstash_redis, name, primaryRegion, planId: upstash_redis_fixed_1gb})withpublicUrlas the Redis URL,addOn(name:) { id name publicUrl primaryRegion addOnPlan { id } }for the retry and for the plan and region the Autoredisstep records,deleteAddOn(input: {name}) { deletedAddOnName }(flyctlredis destroy), and the exec body'sstdinfield (the Redis ping passes the URL on stdin tonode -ein the web machine); that a volume left by a destroyed machine can be mounted by a new machine of the same app. - Metrics: the Prometheus series and labels
metrics.tsqueries (fly_instance_memory_mem_availablewith one sample per 15-second scrape,fly_edge_data_out,fly_edge_http_responses_count,fly_volume_used_pctwith avolumelabel, theapplabel on each) and that a read-only token works onapi.fly.io/prometheus/<org>. Spend computed from them should be compared with one real invoice before the Auto cap is sold.
Docker: the Engine API calls follow v1.41+; the exec start response is read as raw text and the exit code from GET /exec/{id}/json. Kubernetes: manifests are plain apps/v1 / networking.k8s.io/v1; kubectl is shelled out to and was not run here. Railway's railway.json schema and Render's render.yaml keys (runtime: image, disk, generateValue) are from memory of their documentation.
Spillway 0.1.0 · Edit this page on GitHub