Markdown for retrieval

Most systems that feed web pages to a language model do not hand it the whole page. They cut the page into chunks of a few hundred to a couple of thousand characters, store the chunks, and at question time retrieve the few that look most like the question. That is retrieval-augmented generation, and it changes what good Markdown looks like.

Markdown written for a person keeps every heading and every table, because a person reads a section at a time and scans a table by its columns. A chunker does neither. It cuts wherever its size limit falls, often at a heading, and a fact that was spread across a heading, a line and a table cell ends up in pieces that are retrieved separately or not at all. A price ends up in a chunk that never says which product it belongs to. A row from the middle of a table arrives without its column names.

optimiseForRAG shapes the Markdown so that a chunk holds whole facts:

{
  "url": "https://www.anthropic.com/pricing",
  "optimiseForRAG": true,
  "linksEnabled": false,
  "wait": 4000
}

What you see in the result is Markdown with fewer headings, no Markdown tables, and the same words. A product's name stays with its figures, and each row of a table carries the names of its columns, so a chunk cut anywhere still says what it is about. Code samples are left as they are. report.optimiseForRAG says whether the shaping was applied, and the cache keeps the shaped and plain reads of a page apart, so switching it on for one call never hands you a shaped page on the next.

When to turn it on

If the Markdown goes into a chunker and an index, turn it on. Retrieval improves, and nothing you would ask a retriever for is lost.

If people read the Markdown, or your code parses its headings and tables, leave it off. There are fewer headings, so a table of contents built from the Markdown is shorter, and there are no Markdown tables, so a renderer that draws tables has nothing to draw.

linksEnabled: false is a good companion. Addresses take up room in a chunk and match nothing anyone asks.