Fetch a page

POST https://api.quorumtech.ch/kerf/v1/extract/url

Fetch a page and convert it to Markdown.

Fetches the address you give it and returns Markdown whose reading order follows the page's layout rather than the order of the markup: a two-column article is read column by column, a table arrives as rows, and navigation, cookie notices and footers are dropped.

The answer is a job. Most pages are ready in a second or two, so wait holds the request open for up to that many milliseconds and returns the finished extraction directly; past the wait you get a 202 with an id to collect from GET /kerf/v1/jobs/{jobId}. Where the service holds a short cache of pages it fetched recently, a hit answers 200 with cached: true and no job to collect.

A page that only renders in a browser is fetched in one, which is slower. You do not choose and you are not charged differently: a page costs a tenth of a credit whichever way it is fetched.

The tenth of a credit is held when the job is accepted and charged when you collect the page (X-Credits-Reserved on the 202, X-Credits-Charged on the collection). A job that comes back with no page costs nothing. A page answered inline, because wait covered it or it came from cache, is charged on the spot.

The call to start with. Send an address, read Markdown.

Set wait to the milliseconds you are willing to hold the request open. Most pages finish inside two seconds, so a wait of a few thousand usually means one call and no job to follow. Past the wait you get a 202 carrying an id; collect it from GET /kerf/v1/jobs/{jobId} when it is ready, or watch .../events. Collecting is free.

The tenth of a credit is held when the job is accepted and charged when you collect the page, so a fetch that comes back with nothing costs nothing. A page the request itself carries, because wait covered it or the cache did, is charged there and then.

Leave mode alone unless you have a reason. auto fetches the page directly and opens a browser only when what came back is a shell rather than an article, and the price is the same either way, so the default is both the fastest common case and the most reliable one.

Two options exist for particular work. location fetches the page from a country, for prices and terms as a visitor there sees them; optimiseForRAG shapes the Markdown for a chunk-and-retrieve reader. The guides on fetching from a country and Markdown for retrieval say when each is worth turning on.

The service may answer from a short cache of pages it fetched recently, as a 200 carrying cached: true and no job id. It is charged like any other page: the cache is there to spare the site, not to change your bill. It is off on this deployment today, so every call is a fresh fetch.

{
  "url": "https://example.com/",
  "wait": 4000
}

Request

The address, and the few choices that change what comes back. The defaults suit reading a page: the fetch picks its own path, links are kept, images are not, and the site's robots.txt is not consulted.

A JSON body with these fields:

| Field | Type | Required | Description | | --- | --- | --- | --- | | url | string | yes | The page to fetch. http or https, and a public address. | | wait | integer | no | Milliseconds to hold the request open for a page that finishes quickly, so a short fetch needs no second call. Past it you get a 202 with a job to collect. Ignored by /kerf/v1/render, which always waits. | | mode | auto or fast or browser | no | auto fetches the page directly and falls back to a browser when what came back is a shell rather than the article. fast never opens a browser and fails instead. browser always opens one. The price is the same for all three. Default "auto". | | furniture | drop or end | no | Navigation, cookie notices, share bars and footers: dropped, or kept and moved to the end of the Markdown where they do not interrupt the reading. Default "drop". | | hidden | include or drop | no | Text the page holds but does not show yet: inactive tabs, closed accordions, collapsed panels. Included, it is written after the visible text under a <!-- folded --> comment, which is where a pricing page keeps its yearly prices. drop leaves it out. Default "include". | | linksEnabled | boolean | no | Keep links as Markdown links. Off, the text stays and the addresses go. Default true. | | imagesEnabled | boolean | no | Include every image with an address. Off, only images carrying a description are kept, as alt text. Default false. | | followRobots | boolean | no | Consult the site's robots.txt before fetching, and refuse the page with robots_disallowed when it disallows it. Off by default because this fetches one page you named rather than crawling a site. Default false. | | location | us or gb or de or ch or in or sg | no | Fetch the page from a country, so prices, currency and terms are the ones a visitor there sees. A region the fleet does not offer answers 400 location_unavailable. When the country cannot be reached the page is fetched from our own address and report.locationFallback says so; with strictLocation it answers 424 location_blocked instead, and nothing is charged. | | strictLocation | boolean | no | With location: fail with location_blocked rather than fall back to our own address. Nothing is charged for a page that fails this way. Default false. | | optimiseForRAG | boolean | no | Shape the Markdown for a reader that cuts it into chunks and retrieves by similarity, so a chunk holds whole facts: a name stays with its figures and a table row keeps its column names. Fewer headings and no Markdown tables come back, so leave it off when people or a table renderer read the result. How the shaping is done changes as the extractor does; depend on the outcome, not on the shape of a line. Default false. | | humanizeEnabled | boolean | no | Render in a browser that moves its cursor the way a person does. Slower, rarely needed, and it implies mode: browser. Cannot be combined with location. Default false. |

Example request

curl -X POST "https://api.quorumtech.ch/kerf/v1/extract/url" \
  -H "Authorization: Bearer $QUORUM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
  "url": "https://example.com/",
  "wait": 4000
}'

Responses

200

The finished extraction, either because wait covered it or because the page was in cache. A cached answer carries cached: true and costs the same as a fresh one.

Every shape below carries these fields:

| Field | Type | Description | | --- | --- | --- | | state | done | | | result | Extraction | One page, converted. | | result.url | string | The address asked for. | | result.finalUrl | string | The address actually read, after any redirects. | | result.status | integer | The HTTP status the site answered with. | | result.title | string | May be null. | | result.markdown | string | The page, in reading order. | | result.report | ExtractionReport | What the fetch did, for logging and for deciding whether to trust a thin result. The fields below are stable; the report also carries diagnostics that change as the extractor does, so read it by name rather than by shape. | | result.report.path | fast or browser | Whether the page was fetched directly or rendered in a browser. | | result.report.words | integer | Words in the Markdown. A page that should be an article and is not is usually a wall rather than an empty page. | | result.report.items | integer | How many blocks the page was read as, counting paragraphs, headings, list items and table rows. | | result.report.tables | integer | Tables recovered, including ones drawn as boxes rather than marked up as tables. | | result.report.images | integer | | | result.report.links | integer | | | result.report.uncertain | boolean | The layout did not settle into one reading order, so the Markdown may not read the way the page does. Browser path only. | | result.report.interstitial | string | A wall the page put up, when it did: what kind, and whether waiting cleared it. Present only when one was met. | | result.report.fastFallback | string | Why the direct read was not served, when the page went to a browser after one: client_rendered, a script_filled: reason, or the direct read's failure. | | result.report.location | string | The region the page was asked from, when one was. | | result.report.locationFallback | object | Present when a location could not be served and the page came from our own address instead. to is direct; reason is refused or thin. | | result.report.locationFallback.to | string | | | result.report.locationFallback.reason | string | | | result.report.optimiseForRAG | boolean | Whether the Markdown was shaped for retrieval. | | result.renderMs | integer | Milliseconds spent fetching and converting. | | result.cached | boolean | Present and true when this came from a recent fetch of the same page rather than a new one. |

Extraction job done

A finished job, with the page.

| Field | Type | Description | | --- | --- | --- | | id | string | | | url | string | | | queuedMs | integer | | | renderMs | integer | | | edge | string | | | attempts | integer | |

Cached extraction

A page fetched recently enough to be answered from cache. There is no job to collect, so there is no id.

| Field | Type | Description | | --- | --- | --- | | cached | true | |

202

The job was accepted. Collect it at the Location header's address.

| Field | Type | Description | | --- | --- | --- | | id | string | | | state | queued or running | | | url | string | | | queuedMs | integer | Milliseconds the job waited before an edge took it. | | renderMs | integer | Null until the job finishes. May be null. | | edge | string | Which edge is fetching it. May be null. | | attempts | integer | |

Errors

The body is { "error": { "code", "message" } }; the full catalogue with what to do about each code is on the Errors page.

| Status | Codes | When | | --- | --- | --- | | 400 | | The request does not match the schema. error.issues names each failing field. | | 401 | | The key is missing, not recognised, or revoked. | | 402 | payment_required, credits_exhausted, remaining, needed | The workspace has no active API subscription (payment_required), or is prepaid and has no credits left for this request (credits_exhausted, with remaining and needed). Manage billing and add credits at https://developer.quorumtech.ch/billing. | | 403 | insufficient_scope, url_not_allowed, robots_disallowed | The key is not scoped to Kerf (insufficient_scope), the address is one this service does not fetch (url_not_allowed: a private or loopback address, or a redirect to one), or the site's robots.txt disallows the page and you asked for it to be followed (robots_disallowed). | | 424 | site_refused, location_blocked, load_failed, timeout | The page could not be fetched, and the site is the reason rather than us: it refused to serve the page (site_refused, which is the site answering 401, 403, 407, 429 or 451, or a wall it would not lower), the country asked for with strictLocation could not serve it (location_blocked), or it would not load (load_failed, timeout). Nothing is charged for a request that answers this, and sending it again unchanged answers the same way. | | 429 | rate_limited, busy | Over the key's requests-per-minute limit (rate_limited), or the fetching fleet's queue is full (busy). Both carry Retry-After; a busy clears in seconds. | | 503 | no_edge_available | No part of the fetching fleet is answering (no_edge_available). Retry shortly. |

Cost

A tenth of a credit ($0.001) a page, whichever way the page is fetched. Held when the job is accepted and charged when you collect the page; a page answered inline or from cache is charged on the spot. A page that never arrives charges nothing.