The open-source marketing CMS
Menu

Site search: specification

Status: approved, 2026-11-10 (Kevin), with the decisions below; being built level by level. This describes how visitor search should grow from keyword matching to search that understands the site, with AI answers as an option, and what each step needs.

How search works today

  • The page. GET /search?q= renders results in the site's shell: a search form, ten results per page with a highlighted snippet, a count and a pager. It works without JavaScript, is cached for 60 seconds per query and is noindex. /es/search searches the Spanish pages. GET /api/search returns the same for headless front ends.
  • What is indexed. One row per entity: the title, and the body built from every text, textarea and rich text field plus the readable text of the page layout (packages/core/src/marketing/search.ts). Taxonomy term labels go to Meilisearch only. The index is updated in the same transaction as every save and publish.
  • The engines.
    • SQLite (default): FTS5 with bm25 ranking, title weighted 5 to 1 over the body, the unicode61 tokenizer with accents folded, and the last word matched as a prefix. There is no stemming, so "running" does not find "run".
    • Postgres: tsvector with the english configuration for every locale, so Spanish or French pages are stemmed as if they were English.
    • Meilisearch (SEARCH_URL): typo tolerance, facets by type and term, one index per site.
  • What is missing.
    • There is no search box block, no header search and no suggestions while typing.
    • There are no synonyms, no "did you mean" outside Meilisearch, no promoted results, and no filters on the public page beyond the API's type, term and locale.
    • Only pages are searchable, not the documents attached to them.
    • Nobody can see what visitors searched for, which searches found nothing, or which results they clicked.
    • All matching is by keyword. A visitor who types "how much does it cost" does not find the page titled "Tuition and fees".

What world-class looks like

The commercial products (Algolia with NeuralSearch and Ask AI, Google's Vertex AI Search, Coveo) and the open-source engines (Meilisearch's hybrid search, Typesense, OpenSearch, Postgres with pgvector) have converged on the same set:

  1. Keyword search done well: stemming per language, typo tolerance, synonyms, sensible ranking, facets, instant suggestions.
  2. Semantic search: content and queries turned into embeddings so meaning matches, not only words, blended with keyword results (hybrid search) so exact terms like a course code still win.
  3. Answers: a short answer written from the site's own pages with links to its sources, plus follow-up questions.
  4. Insight for the team: what people search for, what they do not find, what they click.

On WordPress and Drupal each of these is a separate product: Relevanssi, SearchWP, ElasticPress or Jetpack Search on WordPress; Search API with Solr, plus the AI module's search with a vector database, on Drupal. Each needs its own server, setup and often a subscription. Spillway can ship all four inside the instance, on the SQLite default, using the AI provider the site has already configured. That is the "world-class search at an open-source price" promise: no search server and no per-query subscription unless a site outgrows the defaults.

The plan in three levels

A site picks its level in Site settings → Search. Each level adds to the one before; a site without AI stays at level 1 and loses nothing.

Level 1: keyword search done well (every site, no AI)

For visitors:

  • Search box. A Search block for any page, and Show search in the header in Site settings → Search (on by default), which puts an always-visible search field with a search button in the site header. Themes style it through the brand tokens.
  • Suggestions while typing. After two characters, up to six page titles that match (with their type) and the most searched queries that match, as an accessible combobox (ARIA 1.2: arrow keys, Enter, Escape, announced counts). GET /api/search/suggest?q= answers from the title index in a few milliseconds. Without JavaScript the form submits to /search as today.
  • Better results page.
    • Filters: type, the terms of chosen vocabularies, and date, as GET parameters so filtered results can be linked and work without scripts.
    • Sort: relevance or newest.
    • Rich results: a type badge, the date, and a thumbnail when the page has an image.
    • Did you mean when nothing matches and a close spelling would.
  • Language-aware matching. SQLite gets a stemming tokenizer per locale where one exists (Porter for English, a Snowball stemmer for the main European languages). Postgres uses the text search configuration that matches each entity's locale instead of always english. Meilisearch already handles this.
  • Documents. Text from PDF, Word and plain-text files linked from published pages or set on a file field is indexed with a link to the page that holds them, behind a switch (Search inside documents).

For the team:

  • Synonyms. Pairs and groups such as "tuition, cost, fees, price", applied to every query (the SQL engines expand the query; Meilisearch has its own synonyms setting).
  • Pinned results. For a query or a set of queries, pin up to three pages to the top ("apply" → the Apply page). Shown with a small Recommended label.
  • Ranking. Weights for title, headings, the SEO description and body. Boosts by type ("programs first"), by freshness, and by popularity from the cookieless visit counter. A Preview box shows the results for any query with the current settings before they are saved.
  • Search insights (Analytics → Search): the top queries, queries with no results, queries whose results nobody clicked, and the click-through rate, for 7, 30 or 90 days.
    • Queries are counted per day, normalized (lowercase, trimmed) and stored without any visitor identifier, so this needs no consent.
    • Clicks are counted by a redirect-free beacon in the same cookieless way as the visit counter.
    • Each zero-result query has Add synonym, Pin a page and Create a page next to it.

Level 2: semantic search (with AI configured)

What changes for visitors: the same search box and results page, but results match meaning. "How much does it cost" finds Tuition and fees; "parking for visitors" finds the Campus map page that mentions the visitor lot. Exact words still win: a course code or a name ranks first when it is on a page.

How:

  1. Chunks. Each published entity is split into passages at its headings (about 200 to 400 words each, with the page title and heading kept as context). Documents are chunked the same way.
  2. Embeddings. By default each chunk is embedded by the local model bundled in the image (decided above), so semantic search needs no provider. A site can choose a hosted model instead through the AI layer's new embed() call:
    • OpenAI and Google offer embeddings directly;
    • Anthropic does not, so sites on Anthropic pick OpenAI, Google, Voyage AI or a local model;
    • an OpenAI-compatible endpoint covers local models (Ollama, Text Embeddings Inference), which keeps content and queries on the site's own servers.

    A chunk is re-embedded only when its text changes (by hash), so an edit to one paragraph costs one call.

  3. Storage and lookup.
    • Small sites (up to about 20,000 chunks): vectors live in a search_chunks table and are searched in process by cosine similarity, with no extension, on SQLite or Postgres alike.
    • Large sites: pgvector on Postgres, sqlite-vec on SQLite, or Meilisearch's hybrid search with user-provided vectors.
  4. Hybrid ranking. The keyword engine and the vector lookup each return their top results, merged by reciprocal rank fusion. Pins, boosts and filters apply after the merge. Meaning versus exact words (a slider, default even) shifts the weight; Meilisearch calls this the semantic ratio.
  5. The query. Each search embeds the query once (cached for an hour per normalized query), which costs a fraction of a cent with hosted models and nothing with a local one.

Costs and controls:

  • Turning semantic search on shows the estimated tokens and cost to embed the whole site before it starts, as the importer does.
  • Embedding and query calls count against the instance's monthly AI token cap.
  • When the cap is reached, search falls back to keyword results without an error.

Level 3: answers (opt-in)

What visitors see. On the results page, above the results, an Answer box streams a short answer to questions (queries that read as a question, or anything typed into Ask), written only from this site's pages:

  • Every sentence that states a fact carries a numbered citation linking to the page it came from. The cited pages are listed under the answer.
  • Lists when the question asks for them: "which nursing programs are online?" returns a list built from the matching program pages, using structured fields (a content type's fields, taxonomy terms) as well as text, so the list is complete rather than a guess.
  • Suggestions: two or three follow-up questions and related pages. A visitor can ask up to five follow-ups in a short conversation.
  • Not found, honestly: when the pages do not answer the question, the box says "I couldn't find that on this site", shows the closest results and, if the site set one, a contact link. It never answers from general knowledge.
  • A label says the answer is AI-generated from the site's pages and to check the linked pages; thumbs up and down feed the insights.

An optional Ask block puts the same experience anywhere on a page (a help center, an admissions page) with its own suggested questions, and Show Ask in the header adds it next to the search button.

Guardrails:

  • Sources. Retrieval only reads published content of the types the site allows (Site settings → Search → Use in answers; form submissions, drafts and user data are never searchable).
  • Prompt injection. Page text is passed to the model as quoted sources, never as instructions, and the answer is sanitized Markdown with links only to this site's pages and the cited sources.
  • Limits.
    • per visitor IP: 10 questions a minute, IP not stored;
    • per site: a daily question cap;
    • per answer: output length;
    • the monthly token cap, after which the Answer box is hidden and keyword results remain.
  • Caching. Answers are cached per normalized question and locale, tagged with the entities they cite, so publishing a cited page evicts the answer. Common questions cost one model call per change, not per visitor.
  • Insights. Analytics → Search adds the questions asked, those answered "not found", and the feedback, so the team writes the missing pages. A site agent task can send this weekly.

Privacy. Questions are sent to the configured AI provider. The privacy policy needs a line about it. A local model keeps questions on the site's own servers. No cookies are set, so the consent banner is not involved.

Headless and agents

  • GET /api/public/search, GET /api/public/search/suggest and POST /api/public/ask (server-sent events) give the Astro and Next adapters the same features. @spillway-cms/client gains typed calls.
  • The MCP search_site tool and the site agents use the hybrid search, so agents find content by meaning too.

Data and code

  • AI layer. AiClient.embed(texts) → { vectors, tokens } for OpenAI, Google and OpenAI-compatible providers, plus Voyage AI as an embedding-only provider. New AI tasks embed and answer in Site settings → AI, each with its own provider and model.
  • Search provider. SearchProvider gains suggest() and an optional vectorQuery(). Hybrid fusion lives above the providers, so every engine gets it.
  • Tables (one migration):
    • search_chunks: entity_id, site_id, locale, seq, heading, text, hash, vector (blob), model;
    • search_synonyms;
    • search_pins;
    • search_query_stats: site_id, day, query, count, zero_results, clicks;
    • search_answer_feedback.
  • Indexing. The chunk and embed work runs on the ticker in batches (a queue like cdn_purge), not in the save transaction. The keyword index stays transactional as today.
  • Settings. A search object in the site settings: headerSearch, headerAsk, level, documents, weights, boosts, semanticRatio, answerTypes, dailyQuestionCap, contactHref. Validated in routes/site.ts.

Phases

Phase 1: keyword search done well.

  • The Search block, header search and suggestions;
  • filters, sort and rich results;
  • per-locale stemming and the Postgres configuration fix;
  • synonyms, pinned results, weights and boosts with preview;
  • search insights.

Tests: stemming per locale on both SQL engines, suggestions' keyboard behavior, insights recording no identifiers.

Phase 2: semantic search.

  • the bundled local embedding model, and embed() in the providers;
  • chunking, in-process vectors, hybrid fusion, the cost estimate and the fallback;
  • pgvector and sqlite-vec for large sites, and Meilisearch's hybrid mode.

Tests: fused ranking on a fixture site where meaning and keyword queries each need the other.

Phase 3: answers.

  • The Answer box, the Ask block, lists from structured fields, follow-ups;
  • caching by cited entities, limits, insights for questions.

Tests: answers cite only retrieved pages, "not found" with no sources, injection fixtures, cap fallback.

Phase 4: wider sources. Document indexing, searching across an instance's sites, and the registry and docs searching through the same engine.

Decisions

Decided 2026-11-10 (Kevin):

  1. Semantic search is on by default when AI is configured, with the embedding estimate shown before the first run. It can be turned off per site.
  2. A small local embedding model ships in the image, so semantic search works with no AI provider and no content leaves the server. A hosted embedding model is an option in Site settings → AI.
  3. Header search is an always-visible field with a search button, on by default and switchable per site.
  4. Configurable where appropriate: the level, header search, the Search block's options, weights, boosts, synonyms, pins, the meaning versus exact words balance and the answer settings are all per-site settings.

Still open:

  • Answers on Accorbis plans: a per-site question allowance, or part of the AI credits? Left open for now.

Spillway 0.1.0 · Edit this page on GitHub

Was this page helpful?