Spillway CMSStructured content like Drupal. Page building like Puck. Setup like Ghost. AI that drafts, people who approve.
Menu

The domain importer

Point Spillway at a domain and it brings the site in without a database connection: it crawls the public pages, extracts each one, clusters the pages into content types, converts the bodies into blocks, extracts the theme, keeps every URL and lands the result as drafts with a comparison and a quality report. This page is the reference for how each step works; the admin walkthrough is in getting-started.md and the step-by-step recipe in the guide. The contract is the Importer block of packages/shared/src/index.ts.

Crawl

A job starts with POST /api/imports { url, options? } (permission imports.manage) and the ticker (jobs/ticker.ts) picks it up within a tick; in tests and scripts getImporter(ctx).runJob(id) runs it to completion in-process. The crawler (packages/core/src/importer/runner.ts) works through a frontier stored in import_queue, so a job interrupted by a restart carries on from the undone URLs when the next tick finds it in status crawling. Every fetch goes through the SSRF-guarded transport (forms/http.ts: DNS pinned, private ranges refused) with the user agent SpillwayImporter/<version>.

What it does, in order:

  1. robots.txt is read first (respectRobots, default on). The group for SpillwayImporter wins over *; Allow/Disallow are matched longest-rule-first with * and $ wildcards; a Crawl-delay larger than the job's rateLimitMs replaces it (capped at 30 s); Sitemap: lines seed the sitemap step.
  2. Seeds: the source URL, then every URL of /sitemap.xml (and the sitemaps robots.txt names, /sitemap_index.xml, child sitemaps of an index, up to 50 files and 50,000 URLs) when useSitemap is on.
  3. Fetching runs with 4 workers under a per-host rate limit (rateLimitMs, default 500 ms between request starts; never under IMPORT_MIN_RATE_LIMIT_MS, 100 ms, so a job cannot hammer a host), a 10 MB cap per page, a 2 GB byte budget, a 60 minute time budget and the page cap (maxPages, default 200, at most 5,000). Redirects are followed by hand (5 hops, each re-checked) and every hop is recorded in import_redirect with origin: crawl; the page keeps its requested url and the finalUrl it came from. A 404 or any 4xx/5xx is recorded as a page row with statusCode and error (HTTP 404) and counts as failed. A response that is not HTML (a PDF, an image) becomes an import_asset row instead of a page.
  4. Scope and lists: only links on the source host are followed (followSubdomains adds *.example.org); include and exclude are matched against the path as globs (/news/**, /people/*; a pattern without wildcards is a prefix, so /news covers /news/x) or regexes written re:^/news/\d+ (a re: source may not use nested quantifiers such as (a+)+, backreferences or more than 12 quantifiers, and is at most 200 characters: it runs on every discovered path, so patterns that could backtrack catastrophically are refused with a 400). The source URL is always fetched. Paths with a file extension are never queued as pages.
  5. Normalization and dedupe: URLs are lowercased on the host, stripped of the fragment, tracking parameters (utm_*, fbclid, gclid, ...) and a trailing slash, so one page is fetched once. Two URLs that resolve to one final URL (a redirect, a trailing slash) keep the first page; a page whose <link rel=canonical> names the same path with a different query (/news?tag=campus → /news) is recorded as a variant and skipped, and a canonical that names another path is queued.
  6. Extraction (extract.ts) runs on every HTML page: the main region by landmark (<main>, role=main, #main-content, #content, a single <article>) or the largest text block, with navigation, asides, cookie banners, share bars, search forms and hidden elements removed; the result is made absolute and sanitized through richtext/sanitize.ts into mainHtml, with text and wordCount. From the whole document: the title (site suffix stripped), meta description, canonical, language, Open Graph tags, JSON-LD, published/modified dates (article:* meta, JSON-LD, <time>), the author, headings, links with an internal flag (stylesheets appear with the text [stylesheet] for the theme extractor), images with alt and size, videos (YouTube, Vimeo and the other hosts, <video>), forms with their fields (label, type, required, select options; search forms left out) and breadcrumbs. Images in the main region, the OG image, linked documents and media files are recorded in import_asset as pending; nothing is downloaded during the crawl.
  7. Skeleton hash and URL pattern: skeletonHash is a signature of the main region's structure (the set of chains of classed, id'd or sectioning elements, class tokens normalized, text, media and the inside of body containers ignored), so pages built from one template share it; urlPattern replaces numeric and slug-like segments (/news/2024/09/new-building → /news/{id}/{id}/{slug}). The classifier groups on both.
  8. Log: the last 200 lines are kept in memory and on the job row (GET /api/imports/:id/events).

The raw HTML of every page is stored under <DATA_DIR>/imports/<jobId>/<sha1(url)>.html and served by GET /api/imports/:id/pages/:pid/raw with Content-Security-Policy: sandbox; script-src 'none', so the comparison view can show the original without running anything in it. DELETE /api/imports/:id removes the rows and the directory (landed entities stay). POST /api/imports/:id/recrawl { pageIds? } drops the listed pages and queues their URLs again, or, without ids, starts the whole crawl over. POST /api/imports/:id/cancel stops a running job at its next page.

Assets are downloaded later, one by one, by the landing step through ImporterService.downloadAsset(jobId, assetId): a 50 MB cap, a content-type allow-list (images, PDF and office documents, audio and video, archives, CSV and plain text; HTML and scripts are refused), the MIME type sniffed from the bytes, the file written under the job directory and the row marked downloaded or failed with the reason.

Adapters

After the first page is in, the crawler reads its fingerprints (the generator meta tag, X-Generator/X-Drupal-Cache headers, wp-content/wp-json links, drupal-settings-json) and probes the APIs the job's adapters option allows (default both): /jsonapi for Drupal, /wp-json/ for WordPress. When one answers, the adapter reads what the crawl cannot see and stores it on job.detected (platform, version, generator, apis, types, taxonomies, authors); the page markup adds per-page meta.hints ({ platform, type, id } from Drupal's page-node-type-* body class and rel=shortlink, WordPress's rel=alternate JSON link and single-*/postid-* classes) so the classifier can seed a cluster's type from the source's own type.

  • Drupal (JSON:API): node bundles (node_type names when readable, else the node--* resource links) with the attribute and relationship names of a sample node (body, field_*, minus the internal columns) and the item count; vocabularies with their term names; menu_link_content rows as a tree per menu (internal:/x, entity:node/5 and absolute links resolved, disabled links skipped) into import_menu; the redirect module's rows (redirect--redirect, when exposed) into import_redirect with origin: source; user display names when user--user is readable.
  • WordPress (REST): post types from /wp/v2/types (attachments, blocks, templates and navigation items left out) with the keys of a sample item (title, content, excerpt, author, featured_media, acf.<field> when ACF exposes its fields, the taxonomies attached) and the X-WP-Total count; taxonomies with their terms; authors from /wp/v2/users; menus from /wp/v2/menus and /wp/v2/menu-items when the site exposes them to anonymous requests (core does not, so the navigation usually comes from the crawl). WordPress has no redirect table in core; redirects come from the crawl.

Everything an adapter reads is optional: a 401, a 403 or a missing resource is logged and skipped, and the crawl path stands on its own for any site.

Classification

POST /api/imports/:id/classify (status crawled or later) groups the fetched HTML pages into clusters and proposes a content type per cluster; GET /api/imports/:id/clusters lists them and PUT /api/imports/:id/clusters/:cid records the owner's decision. The code lives in packages/core/src/importer/{cluster,classify,sources}.ts.

Clustering (cluster.ts). Pages are grouped by urlPattern and skeletonHash. Groups with the same pattern whose page shapes are near-identical (a vector of heading outline, image count, forms, breadcrumb depth, word count and whether a date is present; cosine similarity ≥ 0.85) are merged, so an article with one extra sidebar block lands with the other articles. A group needs at least two pages and a pattern with a token (/news/{slug}); everything else (home, about, contact, the listing pages, a pattern with a single page) is pooled into one Pages cluster that maps to the existing page type and is approved by default. Cluster ids are deterministic per job, pattern and skeleton, so running the classification again keeps the ids the owner already decided on: a surviving cluster keeps its decision, target type and (when approved) its edited proposal and label. Labels come from the URL pattern (/academics/programs/{slug} → "Program", /people/{slug} → "Person") with a prefix added when two clusters would share a label.

Proposals (classify.ts). The deterministic pass reads what the pages carry and writes a TypeProposal: title (from the first h1 when most pages have one, else the page title), summary (textarea, meta:description), body (layout, the converted blocks), heroImage (media, image:first), publishDate (date, date:published), author (meta:author) and section (the parent breadcrumb) when at least half of the pages carry them; schema.org types add their fields (Person → jobTitle, email, telephone; Event → startDate, endDate, location). The adapters' per-page hints (meta.hints.type, a Drupal node type or a WordPress post type) name the proposal after the source's own type, using the label job.detected.types declares. The reason says what the pages have in common in plain words; pathPattern is the cluster's static prefix plus {slug} so URLs are kept. A proposal maps onto an existing type (existingTypeKey) when a type with the same key exists (a previous run of this import created it) or when at least 60% of the proposed field keys exist on a type whose path prefix matches; the proposal then carries that type's fields with the sources it can fill (a richtext body takes the plain text). Keys are made unique across the proposals and the existing types.

Every field has a source rule (sources.ts), the same grammar the AI prompt explains and the filler reads: title, h1, meta:<name>, date:published | date:modified, body (type layout only), text, image:first, breadcrumb:<n> (negative from the end, -2 is the parent), selector:<css> (the text of the first match in the main content), og:<property>, jsonld:<property>, url, path. Field types are limited to those an import can fill (text, textarea, richtext, number, boolean, date, select, media, link, json, layout); references, taxonomy and date ranges are added by hand afterwards.

AI refinement. When AI is enabled and the import task has a provider with a key (Site settings → AI, task "Site import"), the proposals go to the model with three sample pages per cluster (title, h1, headings, description, date, author, image count, breadcrumbs, JSON-LD types, a 400-character excerpt) and the existing types; it returns better names, descriptions, field labels and reasons as strict JSON (ai/prompts/import-types.ts). The answer is validated field by field: unknown clusters, field types outside the list, sources that do not parse, invalid keys and existingTypeKey values that name no type are dropped, the title and body fields always survive, and an existing-type mapping only lets labels change. A provider error or unusable JSON leaves the deterministic proposals in place with a note; classification never depends on the model. A job created with options.useAi: false (the wizard's Use AI for type proposals box, --no-ai in the CLI) never calls the model and notes so. The response of classify is { job, clusters, notes, usedAi }.

Decisions. PUT /api/imports/:id/clusters/:cid takes { decision?, label?, targetTypeKey?, mergedIntoId?, proposal? }: approved creates (or maps onto) the type at landing, merged folds the cluster into another approved cluster (mergedIntoId; its pages convert with the target's proposal and land under the target type's pattern, with a 301 from their old path), skipped leaves its pages out, targetTypeKey sends the pages to a type the site already has. An edited proposal is validated like the model's (a title field and matching body/layout pairs are required, sources must parse, existingTypeKey must exist). Audit action import.cluster.update.

AI usage

AI is on by default when the site has a provider for the import task (Settings → AI); the importer shows how much it will use before it spends anything, and each job can turn it off.

Where the importer calls the model. In one place: the type proposals of the classification (refineWithAi in classify.ts). One classification is one import task run that sends every cluster except the one-off pages in a single request, with three sample pages each, so it makes one provider call, or two when the first answer is not valid JSON and the model is asked once more. A classification where every page is a one-off page makes none. Running the classification again spends the same again. Conversion, the theme extraction and the landing make no AI call (alt text is kept from the source, not generated). What grows with the site is the size of that one request and its answer, not the number of calls.

The estimate. The AI layer meters tokens, not money (ai_log and the monthly token cap under Settings → AI), so the estimate is in calls and tokens and the share of the monthly cap; it is never priced. Tokens are counted at four characters per token, input and output together.

  • Before the crawl (POST /api/imports/estimate, the New import screen, the first lines of spillway import): the clusters are not known yet, so the importer reads robots.txt and the sitemap the way the crawl will (in scope, the path rules, robots rules; at most 10 sitemap documents, cached for five minutes per source and scope, nothing stored) and counts the URL patterns with a variable part and two or more pages among the first pages up to the cap. That gives the low end of the clusters; the high end allows each pattern to split by page shape and adds room for pages the sitemap does not list. Without a sitemap the range comes from the page cap alone (one cluster up to one per ten pages). The calls are then about 1 (2 at most), and the tokens a range from the fixed part of the prompt (the instructions and the site's existing types) plus 700 to 1,500 tokens of request and 150 to 450 tokens of answer per cluster, the answer capped at the task's 6,000-token budget, with the retry counted in the maximum.
  • After the crawl (GET /api/imports/:id/estimate, the Types step, spillway import --classify before it classifies): the importer clusters the crawled pages exactly as the classification will and measures the request it would send, so the clusters are exact and only the length of the answer and the retry are estimated.

Each estimate names the remaining monthly cap (cap.remaining, the cap minus what ai_log counts this month) and sets exceedsCap when the AI will run and the maximum is more than what is left. The wizard then shows a warning and asks for an explicit I understand: start anyway before Start the crawl (and Run classification) can be pressed. The cap is enforced as before: once it is reached the AI layer refuses the call, the classification keeps the heuristic proposals and says so in its notes, and the rest of the import is unaffected.

Conversion

POST /api/imports/:id/convert { clusterIds? } (status classified or later) fills the fields and converts the bodies of every page of an approved (or merged) cluster, scores each page and sets the job to planned; the response is { job, converted, skipped, averageScore, notes }. Pages of pending or skipped clusters are left as crawled (skipped). GET /api/imports/:id/pages/:pid/compare returns { original: { html, text, images, links }, converted: { blocks, html, text }, quality } with the converted page server-rendered through the real block library (resolvers and sanitizing included, no shell). The code is packages/core/src/importer/{convert,fields,report,plan}.ts.

Fields (fields.ts). Every page of a cluster is filled from the same source rules, so the result is consistent across the type. Values are coerced to the field type (dates to ISO 8601, numbers parsed, selects checked against their options, media holds the source image URL until landing swaps it for the media id, link becomes { href }); a source that yields nothing leaves the field out, except the title, which falls back to the page title and then to the last path segment. Selector sources read the raw page's main region (classes intact), not the sanitized extract.

Blocks (convert.ts). The converter walks the main region of the fetched HTML (the raw file on disk, picked with the extractor's landmarks and boilerplate rules; the sanitized mainHtml when the file is gone) in document order and segments it into the existing blocks:

  • a Hero from the opening h1 when it sits in a hero/banner wrapper or has a large image (≥ 600 px wide, or hero-classed) before or beside it, with the first short paragraph as subtitle and up to two links as buttons; an article whose image follows its title keeps a Heading (h1) and an Image;
  • headings, paragraphs, lists, tables, blockquotes, code and definition lists into Text blocks in HTML mode, sanitized by richtext/sanitize.ts (a later h1 becomes an h2, details/summary and FAQ lists become headed sections: there is no accordion block);
  • standalone images, figures and linked images into Image (alt and caption kept); runs of three or more, and image galleries, into Columns of images, three per row;
  • repeated groups of link + image/heading + short text (a list of teasers, a grid of cards) into Cards; a plain list of links stays rich text;
  • <iframe> into Embed only when the host is on the site's embed allow-list (DEFAULT_EMBED_HOSTS or SiteSettings.embedAllowedHosts); other frames become a link with the issue embed_dropped; <video>, <audio> and <object> become links (media_as_link);
  • <form> into a Form placeholder with an empty formKey, the form's heading as title and the extracted fields (name, type, label, required) in the intro and on the page's forms, with the issue form_placeholder; forms the crawler recorded that the region lost are appended the same way;
  • a block of one heading, at most two sentences and one button-like link into CallToAction; a button-like link on its own into Button;
  • scripts, styles, navigation, templates and hidden elements are removed (an external script is noted as script_dropped); anything else is unwrapped and its text flows into the surrounding rich text. Nothing is passed through raw.

Links to the source host become site-relative paths (https://www.example.edu/about/ → /about), other links stay absolute; image URLs stay absolute until landing re-links them. Each top-level block gets a fragment hash (its type and props without ids); landing turns fragments shared by three or more pages into shared blocks.

Quality (report.ts). Every converted page gets an ImportQuality: textRetained (the share of the source's words, measured at text-node boundaries and without stop words, present in the blocks), imagesRetained and linksRetained (source URLs still referenced, media re-links counted), issues with a code and severity (empty_page, missing_title, text_lost, images_lost, links_lost, missing_alt, broken_internal_link for links to paths the crawl did not fetch, plus the converter's embed_dropped, script_dropped, form_placeholder, media_as_link) and a score from 0 to 100 (100 minus the weighted losses, 10 per error and 3 per warning). The admin's quality filter and the report's below70 count read it.

Landing

POST /api/imports/:id/land { pageIds?, clusterIds?, publish? } (status planned or landed) turns the plan into site content and returns the ImportReport plus a summary (entities, types, assets, redirects, menus, sharedBlocks, workstreamId, notes); GET /api/imports/:id/report returns the report alone. The code is packages/core/src/importer/land.ts.

In order:

  1. Types. For every approved proposal (merged clusters follow their target) the type is created from the proposal (key, name, description, pathPattern with {slug}, the fields with their types; only the title is required, because an import cannot promise every page carries every field) or, when a type with that key or existingTypeKey exists, the fields it lacks are added. Audit type.create / type.update.
  2. Workstream. With options.workstream (the default) the entities land inside a workstream keyed import-<job id prefix> and labeled after the host, created on the first landing and reused by later ones (job.workstreamId). Review the drafts there, preview the site as it would look, and merge to publish.
  3. Assets. Every image a landing page references (block props and rich text), every media field value and every linked document on the source host (PDF and office files) is downloaded through ImporterService.downloadAsset (50 MB cap, type allow-list, bytes sniffed), filed in the media library with the alt text the page gave it, and re-linked: Image.src, hero backgrounds, card images, <img src> and <a href> in rich text point at /media/..., media fields hold the media id. A failed download leaves the original URL and a note. Assets already downloaded by an earlier landing are reused.
  4. Shared blocks. Fragments identical on three or more pages (a promo, a repeated video, a card grid) become shared blocks keyed import_<job>_<hash>, and each page references them with a Shared block; a rerun updates them in place.
  5. Entities. One draft per converted page, of the cluster's type, with the filled fields and the blocks in the layout field (or the plain text in a richtext body when the target type has no layout field), at the page's original path: the slug is the path under the type's prefix (/news/story → story, nested paths keep their slashes). A page whose path cannot be kept (a merged cluster under another prefix, a path already taken) lands where the pattern puts it and gets a 301 from the old path, except the site root: the home page lands at /home (with a suffix when the starter page holds it) and is assigned as the home page under Site settings after the merge. The entity id is stored on the page (entityId), so a rerun updates the entity instead of creating another; an entity lost from the row is found again by path and type.
  6. Redirects and menus. The source's own redirects and the ones the crawl met become site redirects (a Drupal /node/<id> target resolves to the page carrying that id), each created once. A redirect whose target is another host (an absolute http(s):// URL or //host after the source's own URLs have become paths) is dropped by default (options.offsiteRedirects: 'drop'): it is counted in summary.redirectsDropped with a note, and nothing on the new site sends visitors off-site without an editor deciding so. offsiteRedirects: 'keep' lands them like any editor-made redirect (each with an audit row); the Land step says which applies. The source's menus (from the adapters, else the crawled navigation) are created as import-<key> menus (Imported: Main) with items linking the landed entities by id where a path matches and plain hrefs elsewhere, two levels deep at most; a rerun replaces the items.
  7. Status and report. The job becomes landed, stats.landed and stats.assets are updated, the audit row import.land carries the summary, and the events log lists what happened and every note.

Nothing is published. publish: true publishes the landed entities only when the job lands on the main line (options.workstream: false) and the caller has content.publish; inside a workstream the flag is ignored with a note, and the merge is the publication. Running the landing again is safe: types, redirects, menus and shared blocks are matched by key or path, entities by id, assets by source URL.

Report (ImportReport). pages counts the job's pages by status (crawled, converted, landed, redirect, failed, error) and by type; quality holds the average score, the number of pages below 70 and the issues by code; redirects and menus count what exists on the site for this job; assets the downloaded and failed rows; theme whether an extract exists and how many tokens it holds; durationMs the time from the crawl start to the last update.

Theme

The theme step reads the stylesheets the crawled pages link to and turns what it finds into theme tokens for the brand draft (the ones under Brand, see theming.md). It is deterministic (no AI) and runs in a few seconds.

What it reads. From up to 30 of the fetched pages (shallow pages first) it collects <link rel="stylesheet"> targets, <style> blocks, Google Fonts links, <link rel="icon">, og:image and the logo candidate (an <img> or inline <svg> in the header whose src, alt, class or id says logo or brand; og:image as the fallback). Stylesheets are fetched through the SSRF guard with the importer's user agent, at most 10 sheets of 2 MB each, and only from the site's own hosts or the CDN hosts the pages preconnect to or load stylesheets from; @import is followed within the same caps. Dark-mode (prefers-color-scheme: dark) and print rules are ignored so they cannot shadow the default theme.

How it maps. A small CSS tokenizer (no dependency) parses rules, nested rules, conditional groups, @font-face and @import, then:

  • custom properties on :root, html and body are resolved (var() chains included) and mapped with the same heuristics as the token import (primary/brand → primary color, text/ink → text, bg/background → background, font-body/font-sans → body font, font-heading/display → heading font, radius, container/max-width); a named property is the strongest evidence;
  • color literals (#hex, rgb(), hsl(), named colors) are counted with the role of the selector they sit in: body background and color, a color and button/.btn background (primary), h1–h6 color, header, footer, card and container backgrounds (surface); grays are told apart from brand colors by chroma;
  • font-family stacks are voted the same way (body → body font, headings → heading font); each family gets a source: google (a Google Fonts link or @import), self-hosted (an @font-face with a src), system or unknown;
  • margin, padding and gap values give the spacing scale (emitted as the custom tokens space-xs … space-4xl), border-radius the corner radius, max-width on containers the content width;
  • custom properties that map onto the extended tokens keep their Spillway names (color-accent, color-secondary, font-size-base, shadow and so on), the site's remaining custom properties become custom tokens (the first 80), and color-surface and color-accent are added from the surface and the second brand color when the site names none.

Every token carries a confidence (0–1) from how dominant the winning vote was and how much evidence there was; the admin shows it as a bar next to each value so you know what to check before applying.

Routes (imports.manage): POST /api/imports/:id/theme/extract runs the extraction and stores it on the job (running it again replaces the earlier extract); GET /api/imports/:id/theme returns it; POST /api/imports/:id/theme/apply with { tokens?, custom?, logoAsMedia? } merges the given tokens (or the extracted ones when omitted) into the brand draft, started from the published brand when the site has no draft, keeping the draft's custom tokens; a value the draft would refuse (a color that is not a CSS color) is left out. It writes the audit rows brand.draft_save and import.theme_apply. The live theme does not change, so no cache is purged: review the draft under Structure → Brand and publish it there. With logoAsMedia: true the logo is fetched (5 MB cap, must sniff as an image), filed in the media library and set as the draft's Logo; the response's logoMediaId names it. Every import goes through review this way, like Import brand, which reads one page of a site (no crawl) into the same draft. The response type is ImportThemeResult: the ImportThemeExtract of the contract plus confidence, spacing, radii, warnings, extractedAt, applied, appliedAt and logoMediaId.

Limits and failure modes. Stylesheets from other hosts, private addresses, HTML error pages served as CSS and sheets past the caps are skipped with a warning in the result (and a line in the job's events log); with no readable stylesheet the tokens are the defaults with confidence 0. Inline SVG logos are kept as a data: URL only up to 48 KB and never with a <script> inside.

Admin

Structure → Import lists the site's import jobs (source, status, pages fetched/discovered, types, landed, started) with New import, Open, Cancel (while running) and Delete (removes the crawled pages and files; landed content stays). The list polls every 5 s while a job runs.

New import takes the URL and the crawl options, each explained in place: maximum pages (1–5000, default 200), delay between requests (default 500 ms), respect robots.txt, read the sitemap, land inside a workstream (default on), the Drupal and WordPress adapters (auto-detected; untick to skip one), and under path filters and more the include/exclude path rules (globs or re: regexes, one per line), follow subdomains, keep redirects to other sites (off by default) and the content locale. Under AI usage, next to the start button, Use AI for type proposals (on by default when a provider is configured; unavailable, with the reason, when none is) and the estimate of the AI calls and tokens with the monthly token cap left, updated as the URL and the options change (AI usage); above the cap a warning and I understand: start anyway hold the start button. Start the crawl creates the job and opens it; the runner picks it up within a second.

The job screen is a five-step wizard with a step strip (arrow keys move between steps; every step is reachable and says what it still needs):

  1. Crawl: a progress bar and the counters (discovered, fetched, failed, skipped, assets, bytes), the detected platform and APIs, the events log (polled every 2 s while the job runs) and the page table with status code, type or URL pattern, quality, word count and depth, a search box and a status filter, and Source / Raw links (the raw link serves the fetched HTML under a sandbox CSP). Cancel in the header stops the runner after the current page.
  2. Types: the estimate measured from the crawled pages (the clusters that will go to the model, the calls and tokens, the cap left; the same confirmation above the cap), then Run classification groups the pages into clusters. Each cluster is a card with its label, page count, URL pattern, sample links and the proposed type: name, key, path pattern and the field table (key, label, type, required, source rule), all editable. Approve creates the type at landing (or maps onto the existing type the proposal matched), Map sends the cluster to a type the site already has, Merge folds it into another cluster, Skip leaves its pages out. Approve all accepts every pending proposal.
  3. Convert: Run conversion fills the fields and turns the bodies into blocks for every page of an approved cluster, then lists the pages with their quality score, text/images/links retained and a quality filter (below 70, with errors). A row opens the Compare drawer: the original page in a sandboxed iframe from /raw next to the converted render from /compare, a text view, and the issues list with their codes.
  4. Theme: Extract theme, then the palette swatches (with counts and roles), the fonts with their source, the logo and favicon, and the token table (value, editable; confidence bar). Tick Add the logo to the media library and Apply to brand draft writes the tokens (and the logo, as the draft's logo) to the brand draft; the live theme is untouched until you review and publish the draft under Structure → Brand.
  5. Land: a summary of what will be created (pages, new types, mapped types, assets, workstream, whether off-site redirects are kept or dropped), then Land … as drafts. The import report follows: pages landed, average quality and the count below 70, redirects, menus, assets, theme, duration, pages by type and status, issues by code and the cluster decisions, with buttons to Preview the workstream, open it, and the Review queue.

Everything is keyboard reachable with the admin's components (DataTable, dialogs with focus trap, live regions for progress). The screens need the imports.manage permission (admins).

CLI

spillway import https://www.example.org --max-pages 300 --rate-limit 250
spillway import https://www.example.org --classify --convert --theme --land --json
spillway import https://www.example.org --no-ai --classify

spillway import <url> creates the job on the default site (as the first admin user) and runs the crawl in-process, streaming the events log to the terminal as it goes, then prints the counters. Flags: --max-pages N, --rate-limit ms, --no-robots (ignore robots.txt), --no-sitemap, --include <glob> and --exclude <glob> (repeatable), --locale <tag>, --no-workstream, --keep-offsite-redirects (land the source's redirects to other hosts; dropped by default), --no-ai (keep the type proposals heuristic: the job's useAi is false and no AI call is made). Before the crawl it prints the AI usage estimate and the monthly token cap left, and with --classify it prints the measured estimate again before classifying (in --json output as aiEstimate); a terminal asks for no confirmation, so above the cap it only warns. The later steps run only when asked: --classify runs the classification and approves every proposal (a terminal has no review step; open the admin to decide cluster by cluster), --convert converts the approved clusters, --theme extracts the theme and writes it to the brand draft (publish it under Structure → Brand), --land lands the drafts and prints the report. --json prints the final job (or the report) as JSON on stdout with the progress lines on stderr. The same spillway import <file.json> command still loads an export file: a path that does not start with http:// or https:// is treated as a file.

Operations

One crawl per instance, one instance per job. The ticker starts the oldest queued job when nothing is crawling in its process. Starting a job is an atomic claim: one conditional UPDATE moves the row from queued to crawling and records the instance in import_job.claimed_by (a random id per server process), so several instances sharing one database never crawl the same job twice; the instance that loses the claim moves on. While it crawls, the instance refreshes heartbeat_at every 30 seconds.

Restarts and crashes. A crawling job whose heartbeat is older than five minutes is treated as abandoned: the next tick of any instance (the restarted one included) claims it and resumes from the undone queue rows, logging resuming after a restart with the expired claim's instance id. Until then the job stays crawling and other instances leave it alone, so a crawl that is merely slow is not run twice. An instance that discovers its claim is gone (its row now names another instance) stops at its next status check. A job that crashed mid-crawl therefore resumes within about five minutes of the crash, not at once.

The later steps run once too. Classify, convert and land move the job into classifying, converting or landing with the same kind of conditional update before they read anything; a second concurrent call (a double click, two admins, a retried request) answers 409 import_status naming the status the job is in. Convert and land read the crawled pages in chunks of 200 full rows, so a five-thousand-page job does not sit in memory whole, and land downloads assets four at a time.

Concurrent workers and duplicates. The crawl's four workers may fetch two URLs that resolve to the same final URL at the same moment; the unique (job_id, final_url) index keeps the first page and the other is counted as skipped (the log says duplicate of <path> ... (fetched concurrently)), never as failed.

Security

Every outbound request of the importer goes through the SSRF guard in forms/http.ts: private and internal addresses are refused (set SPILLWAY_ALLOW_PRIVATE_URLS=1 to import an intranet site on purpose), redirects are re-checked per hop, every body is capped and the raw HTML is served back only under a sandbox Content-Security-Policy. The platform adapters follow API links only on the source's own hosts. The parsers the source feeds (HTML, selectors, sitemaps, robots.txt, CSS, include/exclude globs) are linear in their input with capped nesting, and theme token values that would fetch or evaluate (url(), expression(), @import) are dropped when the theme is applied. Nothing the importer writes is published: entities land as drafts, the theme is applied only when a user clicks Apply or passes --theme. The full list of controls is in security.md.

Spillway 0.1.0 · Edit this page on GitHub

Was this page helpful?