Knowledge Base source types

Dataset, website, and file sources for a Knowledge Base: what each accepts, how it is scoped, and which file formats a file source reads.

A Knowledge Base draws on three kinds of source. A dataset source reads documents from a Sanity dataset, a website source crawls from a starting URL, and a file source ingests uploaded documents. One Knowledge Base can combine all three kinds, and it needs at least one source. Dataset and website sources are re-checked on the Knowledge Base's refresh schedule, which you set in Settings > Refresh interval to weekly, monthly, or off. Scheduled refresh depends on your plan; see Context plans and limits. That schedule is one setting for the whole Knowledge Base rather than a setting per source, and uploaded files are never re-checked.

Each source holds documents: one per page for a website source, one per Sanity document for a dataset source, and one per file for a file source. An item longer than about 40,000 characters of text is split into sections, and each section counts as a document. A Knowledge Base holds at most 5,000 documents across all its sources, and your plan counts the source chunks those documents are cut into. See Sources, documents, and source chunks.

Documents from a Sanity dataset, selected with a complete GROQ query such as *[_type == "article"]. Add a projection to control what each document carries, as in *[_type == "article"]{title, body}. Every field the query returns counts toward your source chunks, so a narrow projection uses fewer, and following references with -> copies the referenced documents' fields into each document. The query has to select documents from the dataset, so a bare filter such as _type == "article" is rejected. Start with a narrow query; you can widen it later.

A Knowledge Base binds one dataset. To point at a different dataset, remove the dataset source and add a new one; the query itself stays editable. Website and file sources can be added repeatedly. A dataset source reads published documents only, and a query that matches nothing is rejected. Connecting a dataset source takes both of these:

• A role on the source project that can create datasets. Among the default roles, that means Administrator or Developer.
• Unrestricted read access to the dataset itself. A role whose read grant is filtered to a subset of documents is not enough.

The import fails when the Sanity documents its query matches, plus the documents already in the Knowledge Base from other sources, would exceed 5,000. Narrow the query, or remove other sources, to make room.

A crawl starting from a URL. Crawls respect robots.txt. Use the most specific URL you can. A docs section is usually more useful than a whole domain.

Uploaded files never re-sync. To update one, delete the import and upload the new version; deleting an import removes every document it produced. Archives are expanded and their contents ingested individually, each by its own format.

A file source reads the formats below, subject to the size limits shown.

Around forty further text formats are ingested verbatim in the same way, so the table is not the full list of what a file source reads. A file in a format with no handler is not rejected: it is recorded with its metadata, but its contents are not read. Every upload is capped at 5 GiB, and for the formats listed above the per-format limit applies first.

If the material changes regularly, connect it as a dataset or website source rather than uploading a snapshot. Uploads are right for material that is fixed, or that has no addressable source, such as a signed policy PDF or an export from a system Context cannot reach.

• Sources, documents, and source chunks. How each source is processed into documents and source chunks.
• Quick start: build a Knowledge Base and ask it a question. Add a source and run the first build.
• Keep a Knowledge Base current. Which sources refresh, and how often.