Import a website

The importer brings an existing site into Spillway from its public pages alone: no database dump, no credentials. It crawls, proposes content types, converts the pages to blocks, extracts the theme and lands everything as drafts with redirects, so the old URLs keep working when you go live.

Before you start: an admin account (the imports.manage permission), and the address of the site. The crawler respects robots.txt and waits between requests; a 300-page site takes a few minutes at the default 500 ms delay. For a site on a private network set SPILLWAY_ALLOW_PRIVATE_URLS=1 on the server first.

Steps

In the admin, open Structure → Import and choose New import. Enter the site's URL (a path narrows the crawl to that part of the site). Leave Maximum pages at 200 for a first run and keep Land inside a workstream on. Read AI usage next to the start button: how many AI calls and tokens the type proposals will use and how much of the monthly AI token cap is left; untick Use AI for type proposals to keep them heuristic. Start the crawl.
Watch the Crawl step: the counters and the log update every two seconds; the page table fills with each page's status and, later, its type and quality. Open Raw on a page to see exactly what was fetched. Wait for the status badge to read Crawled.
On Types, run the classification. Read each cluster card: the sample links tell you what the pages are, the proposal names the type and lists the fields with the rule that fills each one (title, meta:description, date:published, selector:.summary, body…). Fix a name or a key, remove a field you do not want, then Approve. Map a cluster to a type you already have when it matches (news onto post, for example), Merge two clusters that are the same thing, Skip search results and the like. Approve all when the proposals look right.
On Convert, run the conversion. Sort by quality and open Compare on the lowest pages: the original and the converted render sit side by side with the issues underneath. A low score usually means a sidebar or a form that did not convert; go back to Types and refine the field rules, then convert again.
On Theme, extract the theme, check the swatches, the fonts and the logo, correct any token with a low confidence bar and Apply to brand draft. Then review the draft under Structure → Brand and Publish it there (or later, after landing).
On Land, read the summary and Land … as drafts. The report lists what was created. Preview the workstream to browse the whole imported site at its original paths; publish from the Review queue page by page, or merge the workstream to publish everything at once.

From a terminal, the same run without the review step is:

spillway import https://www.example.org --max-pages 300
spillway import https://www.example.org --classify --convert --theme --land

The first command crawls and stops; the second approves every proposal, converts, writes the theme to the brand draft and lands the drafts, printing the report. Add --json to get the report as JSON.

Check the result

Structure → Import shows the job as Landed with the page count; its report names a workstream.
The workstream preview (/_preview/ws/<id>/) answers the old paths, for example /news/<slug>, with the converted pages.
Marketing → Redirects lists a 301 for every URL whose path changed.
Brand shows the extracted tokens in the draft; once you publish it, GET /api/theme/tokens.css shows them. To take only a site's look (one page, no crawl), use Brand → Import brand → Website instead (theming.md).
The drafts wait in the Review queue; nothing is published until you say so.

Going further

importer.md: the crawl rules, the adapters for Drupal and WordPress, how clustering and conversion work, the theme heuristics and the CLI flags.
workstreams.md to review and merge the imported site as one change.
theming.md to refine the tokens after the import.

Spillway 0.1.0 · Edit this page on GitHub