Fetch a page

POST https://api.quorumtech.ch/kerf/v1/extract/url

Fetch a page and convert it to Markdown.

Fetches the address you give it and returns Markdown whose reading order follows the page's layout rather than the order of the markup: a two-column article is read column by column, a table arrives as rows, and navigation, cookie notices and footers are dropped.

The answer is a job. Most pages are ready in a second or two, so wait holds the request open for up to that many milliseconds and returns the finished extraction directly; past the wait you get a 202 with an id to collect from GET /kerf/v1/jobs/{jobId}. Where the service holds a short cache of pages it fetched recently, a hit answers 200 with cached: true and no job to collect.

A page that only renders in a browser is fetched in one, which is slower. You do not choose and you are not charged differently: a page costs a tenth of a credit whichever way it is fetched.

The tenth of a credit is held when the job is accepted and charged when you collect the page (X-Credits-Reserved on the 202, X-Credits-Charged on the collection). A job that comes back with no page costs nothing. A page answered inline, because wait covered it or it came from cache, is charged on the spot.

The call to start with. Send an address, read Markdown.

Set wait to the milliseconds you are willing to hold the request open. Most pages finish inside two seconds, so a wait of a few thousand usually means one call and no job to follow. Past the wait you get a 202 carrying an id; collect it from GET /kerf/v1/jobs/{jobId} when it is ready, or watch .../events. Collecting is free.

The tenth of a credit is held when the job is accepted and charged when you collect the page, so a fetch that comes back with nothing costs nothing. A page the request itself carries, because wait covered it or the cache did, is charged there and then.

Leave mode alone unless you have a reason. auto fetches the page directly and opens a browser only when what came back is a shell rather than an article, and the price is the same either way, so the default is both the fastest common case and the most reliable one.

The service may answer from a short cache of pages it fetched recently, as a 200 carrying cached: true and no job id. It is charged like any other page: the cache is there to spare the site, not to change your bill. It is off on this deployment today, so every call is a fresh fetch.

{
  "url": "https://example.com/",
  "wait": 4000
}

Request

The address, and the few choices that change what comes back. The defaults suit reading a page: the fetch picks its own path, links are kept, images are not, and the site's robots.txt is not consulted.

A JSON body with these fields:

| Field | Type | Required | Description | | --- | --- | --- | --- | | url | string | yes | The page to fetch. http or https, and a public address. | | wait | integer | no | Milliseconds to hold the request open for a page that finishes quickly, so a short fetch needs no second call. Past it you get a 202 with a job to collect. Ignored by /kerf/v1/render, which always waits. | | mode | auto or fast or browser | no | auto fetches the page directly and falls back to a browser when what came back is a shell rather than the article. fast never opens a browser and fails instead. browser always opens one. The price is the same for all three. Default "auto". | | furniture | drop or end | no | Navigation, cookie notices, share bars and footers: dropped, or kept and moved to the end of the Markdown where they do not interrupt the reading. Default "drop". | | hidden | drop or include | no | Text the page hides from a reader: dropped, or included. Including it picks up content behind tabs and accordions, and also the text of menus that were never meant to be read. Default "drop". | | linksEnabled | boolean | no | Keep links as Markdown links. Off, the text stays and the addresses go. Default true. | | imagesEnabled | boolean | no | Include every image with an address. Off, only images carrying a description are kept, as alt text. Default false. | | followRobots | boolean | no | Consult the site's robots.txt before fetching, and refuse the page with robots_disallowed when it disallows it. Off by default because this fetches one page you named rather than crawling a site. Default false. |

Example request

curl -X POST "https://api.quorumtech.ch/kerf/v1/extract/url" \
  -H "Authorization: Bearer $QUORUM_API_KEY" \
  -H "content-type: application/json" \
  -d '{
  "url": "https://example.com/",
  "wait": 4000
}'

Responses

200

The finished extraction, either because wait covered it or because the page was in cache. A cached answer carries cached: true and costs the same as a fresh one.

Every shape below carries these fields:

| Field | Type | Description | | --- | --- | --- | | state | done | | | result | Extraction | One page, converted. | | result.url | string | The address asked for. | | result.finalUrl | string | The address actually read, after any redirects. | | result.status | integer | The HTTP status the site answered with. | | result.title | string | May be null. | | result.markdown | string | The page, in reading order. | | result.report | ExtractionReport | What the fetch did, for logging and for deciding whether to trust a thin result. The fields below are stable; the report also carries diagnostics that change as the extractor does, so read it by name rather than by shape. | | result.report.path | fast or browser | Whether the page was fetched directly or rendered in a browser. | | result.report.words | integer | Words in the Markdown. A page that should be an article and is not is usually a wall rather than an empty page. | | result.report.items | integer | How many blocks the page was read as, counting paragraphs, headings, list items and table rows. | | result.report.tables | integer | Tables recovered, including ones drawn as boxes rather than marked up as tables. | | result.report.images | integer | | | result.report.links | integer | | | result.report.uncertain | boolean | The layout did not settle into one reading order, so the Markdown may not read the way the page does. Browser path only. | | result.report.interstitial | string | A wall the page put up, when it did: what kind, and whether waiting cleared it. Present only when one was met. | | result.renderMs | integer | Milliseconds spent fetching and converting. | | result.cached | boolean | Present and true when this came from a recent fetch of the same page rather than a new one. |

Extraction job done

A finished job, with the page.

| Field | Type | Description | | --- | --- | --- | | id | string | | | url | string | | | queuedMs | integer | | | renderMs | integer | | | edge | string | | | attempts | integer | |

Cached extraction

A page fetched recently enough to be answered from cache. There is no job to collect, so there is no id.

| Field | Type | Description | | --- | --- | --- | | cached | true | |

202

The job was accepted. Collect it at the Location header's address.

| Field | Type | Description | | --- | --- | --- | | id | string | | | state | queued or running | | | url | string | | | queuedMs | integer | Milliseconds the job waited before an edge took it. | | renderMs | integer | Null until the job finishes. May be null. | | edge | string | Which edge is fetching it. May be null. | | attempts | integer | |

Errors

The body is { "error": { "code", "message" } }; the full catalogue with what to do about each code is on the Errors page.

| Status | Codes | When | | --- | --- | --- | | 400 | | The request does not match the schema. error.issues names each failing field. | | 401 | | The key is missing, not recognised, or revoked. | | 402 | payment_required, credits_exhausted, remaining, needed | The workspace has no active API subscription (payment_required), or is prepaid and has no credits left for this request (credits_exhausted, with remaining and needed). Manage billing and add credits at https://developer.quorumtech.ch/billing. | | 403 | insufficient_scope, url_not_allowed, robots_disallowed | The key is not scoped to Kerf (insufficient_scope), the address is one this service does not fetch (url_not_allowed: a private or loopback address, or a redirect to one), or the site's robots.txt disallows the page and you asked for it to be followed (robots_disallowed). | | 429 | rate_limited, busy | Over the key's requests-per-minute limit (rate_limited), or the fetching fleet's queue is full (busy). Both carry Retry-After; a busy clears in seconds. | | 502 | site_refused, load_failed | The page could not be fetched: the site refused to serve it (site_refused, which is the site answering 401, 403, 407, 429 or 451), or it would not load (load_failed). Nothing is charged for a request that answers this. | | 503 | no_edge_available | No part of the fetching fleet is answering (no_edge_available). Retry shortly. |

Cost

A tenth of a credit ($0.001) a page, whichever way the page is fetched. Held when the job is accepted and charged when you collect the page; a page answered inline or from cache is charged on the spot. A page that never arrives charges nothing.