Fetching a page from a country

A site answers the address that asks. Ask for a pricing page from Germany and you get euros and EU terms; ask from the United States and you get dollars and a different set of terms. Our servers are in Europe, so without a location that is the page you get. For a lot of work it does not matter. For prices, availability, legal terms and anything with a country picker, it does.

location fetches the page through an exit in the country you name:

{
  "url": "https://slack.com/pricing",
  "location": "us",
  "wait": 4000
}

The regions available today are us, gb, de, ch, in and sg. Send one the fleet does not offer and the fetch is refused with 400 location_unavailable. Both fetch paths honour it: a page read directly is read from that country, and a page that needs a browser is opened in a browser that sits behind that country's exit, with its clock set to match. report.location in the answer names the region.

The price is the same. A page is a tenth of a credit wherever it is fetched from.

When the country cannot be reached

Two things go wrong, and Kerf handles both the same way.

Some sites refuse the addresses that regional exits use, because those addresses are shared and the site has seen them before. Kerf asks twice, the second time from a fresh address in the same country. If both are refused, the page is fetched from our own address instead and served with report.locationFallback set to { "to": "direct", "reason": "refused" }.

Some sites answer a regional address with a stub instead of a refusal: a few hundred characters of scaffolding where the direct page is a real one. Kerf compares the two, serves the fuller page, and says so with reason: "thin".

Either way you get a page, and the report tells you it is not the country's page. Check report.locationFallback when the country is the reason you asked.

When only the country's page will do

Set strictLocation to true and Kerf will not fall back. A page that the country cannot serve fails with 424 location_blocked, on the call itself if you waited for the page and on the job otherwise, and nothing is charged. It is not worth retrying as it is: the site refused that country twice. Drop strictLocation to take our own address's page, or drop location. Use this when a page from the wrong country would be worse than no page: a price you are going to quote, a term you are going to act on.

strictLocation needs a location to be strict about, and the two are refused together with humanizeEnabled, since a humanised browser is not a regional one.

What to expect on speed

A regional fetch adds the round trip to the exit. Direct reads still finish in about a second. Browser renders through a regional exit are slower than local ones, since every asset the page loads travels the same way; allow ten to twenty seconds rather than five to ten. Kerf's memory of which sites need a browser applies to regional fetches too, so a site you fetch often settles to whichever path it needs.