Pages as Markdown

Give Kerf an address and it gives you back the page as Markdown. What makes it different from a converter is the order: the text comes out in the order a person reads the page, not the order the markup happens to be in. A two-column article is read column by column rather than line by line across the gutter. A table arrives as rows. Navigation, cookie notices, share bars and footers are dropped rather than mixed into the text.

Base URL: https://api.quorumtech.ch. A key needs the kerf scope.

The first call

curl -X POST "https://api.quorumtech.ch/kerf/v1/extract/url" \
  -H "Authorization: Bearer $QUORUM_API_KEY" \
  -H "content-type: application/json" \
  -d '{ "url": "https://example.com/article", "wait": 4000 }'

wait holds the request open for a page that finishes quickly, which most do. If the page is ready inside it you get the extraction; if not you get a 202 with an id, and GET /kerf/v1/jobs/{id} collects it when it is done. Collecting is free, however many times you do it.

If you would rather not handle a job at all, POST /kerf/v1/render does the same work and holds the request open until the page is finished. It is the simplest call to write and the one most likely to meet a client timeout on a slow page.

Two paths, one price

Most pages are served as HTML and are fetched directly, which takes about a second. Some pages are a shell that assembles itself in the browser, and those are fetched in a real browser instead, which takes several seconds and costs us roughly twenty times as much.

You do not have to know which is which. mode defaults to auto: the direct fetch is tried, and if what came back is a shell rather than an article the page is opened in a browser. report.path in the answer says which way it went.

A page costs a tenth of a credit either way, which is $0.001 at $1.00 per thousand pages. One price is the whole point: what a page costs you does not depend on how a site chose to build itself, and your bill does not move when a site changes its front end. Set mode to fast if you would rather a browser page failed than took the time, or browser to force one.

When you pay

The tenth of a credit is held when the job is accepted and charged when you collect the page. The 202 says X-Credits-Reserved: 0.1 and the collection that first sees the job finish says X-Credits-Charged: 0.1. Collecting it again is free, and so are the progress and event routes.

A fetch that comes back with nothing costs nothing: a job that failed settles at zero when you collect it. A job nobody ever collects settles at its hold a day later, because the page was fetched whether or not anyone read it.

POST /kerf/v1/render has the page by the time it answers, so it charges there and then, and a refusal charges nothing.

What comes back

{
  "id": "6f1ec0d2-3f44-4a1f-9c1e-2b0f4f4d9d2a",
  "state": "done",
  "url": "https://example.com/article",
  "result": {
    "url": "https://example.com/article",
    "finalUrl": "https://example.com/article",
    "status": 200,
    "title": "An example article",
    "markdown": "# An example article\n\n...",
    "report": { "path": "fast", "words": 812, "items": 34, "tables": 1, "images": 0, "links": 12 },
    "renderMs": 940
  }
}

markdown is the page. report says what the fetch did: which path it took, how much text came out, how many blocks and tables were recovered. Two fields are worth reading in code:

• report.words is your check that a page actually arrived. A news article that comes back as forty words is a wall or a paywall, not a short article.
• report.uncertain (browser path) means the layout did not settle into one reading order. The Markdown is still the page, but its order may not be the one you would read.

Read the report by field name. It also carries diagnostics that change as the extractor does.

Options

| Option | Default | What it does | | --- | --- | --- | | mode | auto | auto, fast or browser, as above. | | furniture | drop | Navigation, cookie notices and footers: dropped, or kept and moved to the end. | | hidden | drop | Text the page hides from a reader: dropped, or included. Including it picks up content behind tabs, and also menus nobody was meant to read. | | linksEnabled | true | Keep links as Markdown links. | | imagesEnabled | false | Keep every image with an address. Off, only images carrying a description survive, as alt text. | | followRobots | false | Consult the site's robots.txt first and refuse a page it disallows. |

When a page does not come back

A site can refuse to serve a page to anyone but a person at a browser, and some do. That answers 502 site_refused, on the call itself if you waited for the page and on the collection if you did not, and neither is charged. If you hold the document itself rather than its address, extracting the file is the way through.

The address itself must be public: private and loopback addresses are refused with 403 url_not_allowed, including a redirect that leads to one.

What is kept

The address you send and the Markdown that comes back are held in memory only, for a short cache window, and then forgotten. Nothing is written to a database and nothing is kept after the service restarts. A job is readable only by the workspace that started it: an id from another workspace reads as though it did not exist.

The fetch goes out from our address, not yours, so the site sees Quorum rather than your infrastructure.