Collect an extraction

GET https://api.quorumtech.ch/kerf/v1/jobs/{jobId}

Returns the job: its state while it runs, and the Markdown once it is done. A job belongs to the workspace that started it, and an id from another workspace reads as though it did not exist. Finished jobs are held for a while and then forgotten, and are not kept across a restart of the service, so a 404 means fetch the page again.

This is where the page is paid for: the collection that first sees the job finish settles the tenth of a credit the dispatch held, and carries it in X-Credits-Charged. A job that failed settles at nothing. Every collection after that is free, and so are the progress and event routes. A job nobody ever collects settles at its hold a day later, because the page was fetched whether or not anyone read it.

While the fetch runs the answer carries state: "queued" or "running" with queuedMs, attempts and the edge doing the work. Once it is done it carries state: "done" and result, which holds the Markdown.

Poll every second or so, or watch .../events instead of polling. A finished job is held for a while and then forgotten, and jobs do not survive a restart of the service: a 404 means fetch the page again.

The collection that first sees the job finish is the one charged for the page, and it says so in X-Credits-Charged. A job that failed is settled at nothing, because a page that never arrived is not a page. A job nobody collects settles at its hold a day later: the fetch happened whether or not anyone read it.

A job belongs to the workspace that started it. An id from another workspace answers 404 job_not_found, the same as an id that never existed, which is what it should look like from there.

Request

Path parameters:

| Parameter | Description | | --- | --- | | jobId | The id from the answer to POST /kerf/v1/extract/url, for example 6f1ec0d2-.... |

Example request

curl -X GET "https://api.quorumtech.ch/kerf/v1/jobs/<jobId>" \
  -H "Authorization: Bearer $QUORUM_API_KEY"

Responses

200

The job, with result once state is done.

Every shape below carries these fields:

| Field | Type | Description | | --- | --- | --- | | id | string | | | url | string | | | attempts | integer | |

Extraction job running

A job that has not finished. result is absent.

| Field | Type | Description | | --- | --- | --- | | state | queued or running | | | queuedMs | integer | Milliseconds the job waited before an edge took it. | | renderMs | integer | Null until the job finishes. May be null. | | edge | string | Which edge is fetching it. May be null. |

Extraction job done

A finished job, with the page.

| Field | Type | Description | | --- | --- | --- | | state | done | | | queuedMs | integer | | | renderMs | integer | | | edge | string | | | result | Extraction | One page, converted. | | result.url | string | The address asked for. | | result.finalUrl | string | The address actually read, after any redirects. | | result.status | integer | The HTTP status the site answered with. | | result.title | string | May be null. | | result.markdown | string | The page, in reading order. | | result.report | ExtractionReport | What the fetch did, for logging and for deciding whether to trust a thin result. The fields below are stable; the report also carries diagnostics that change as the extractor does, so read it by name rather than by shape. | | result.report.path | fast or browser | Whether the page was fetched directly or rendered in a browser. | | result.report.words | integer | Words in the Markdown. A page that should be an article and is not is usually a wall rather than an empty page. | | result.report.items | integer | How many blocks the page was read as, counting paragraphs, headings, list items and table rows. | | result.report.tables | integer | Tables recovered, including ones drawn as boxes rather than marked up as tables. | | result.report.images | integer | | | result.report.links | integer | | | result.report.uncertain | boolean | The layout did not settle into one reading order, so the Markdown may not read the way the page does. Browser path only. | | result.report.interstitial | string | A wall the page put up, when it did: what kind, and whether waiting cleared it. Present only when one was met. | | result.renderMs | integer | Milliseconds spent fetching and converting. | | result.cached | boolean | Present and true when this came from a recent fetch of the same page rather than a new one. |

Errors

The body is { "error": { "code", "message" } }; the full catalogue with what to do about each code is on the Errors page.

| Status | Codes | When | | --- | --- | --- | | 401 | | The key is missing, not recognised, or revoked. | | 403 | insufficient_scope, url_not_allowed, robots_disallowed | The key is not scoped to Kerf (insufficient_scope), the address is one this service does not fetch (url_not_allowed: a private or loopback address, or a redirect to one), or the site's robots.txt disallows the page and you asked for it to be followed (robots_disallowed). | | 404 | job_not_found | No extraction with that id (job_not_found). Either it was never started here, it has been forgotten, or it belongs to another workspace, which reads the same way on purpose. | | 429 | rate_limited, claims_rate_limited, claim_budget_exhausted, monthly_cap_reached, cap | Over the key's requests-per-minute limit (rate_limited), its claims-per-minute ceiling (claims_rate_limited), its claims-per-day budget (claim_budget_exhausted), or, for a postpaid workspace, its monthly credit cap (monthly_cap_reached, with cap and resetsAt). Retry-After says when to try again; the daily budget resets at midnight UTC and the cap on the first of the month. | | 502 | | The page could not be fetched. The job carries the reason. |

Cost

This is where a page is paid for: the first collection that sees the job finish settles the tenth of a credit the dispatch held. A failed job settles at nothing, and every collection after the first is free.