Politeness rules
EdgeFetch fetches pages only when a user or agent asks it to. It is not a web-wide crawler, and it isn’t used to train AI models. These rules are always on: they aren’t settings you can turn off.
robots.txt and Content Signals (100% RFC 9309)
Section titled “robots.txt and Content Signals (100% RFC 9309)”robots.txtis checked following RFC 9309 (1 MiBparse cap, UTF-8 BOM handling,AllowandDisallow,*and$wildcards, §2.2.1 group combining, §2.2.2 percent-encoding normalization, §2.3.1.2 5-redirect handling, §2.4Cache-Control: max-ageup to 24h, andCrawl-delay) before the first request and again on every redirect hop.- Temporary
/robots.txtnetwork timeouts or5xxerrors are cached for 60 seconds and reported asblocked: { reason: "robots_unreachable" }(per RFC 9309 §2.3.1.4), distinct from explicit pathDisallowrules (reason: "robots"). - Content Signals: EdgeFetch parses
Content-Signallines (search,ai-input,ai-train) in the matchingUser-agentgroup (inheriting global/*signals when a specific group omits them). Whenai-input=nois set, EdgeFetch treats the page as disallowed and returnsblocked: { reason: "content-signal" }. EdgeFetch never trains models. - Owner-Explicit Partner / Self-Owned Bypass (
ROBOTS_BYPASS_HOSTS):respectRobots: trueis locked on by default (400 robots_requiredif set tofalse). Operators crawling domains they own or hold written partner authorization for can allowlist specific domain suffixes in bothROBOTS_BYPASS_HOSTS(on the Hub) andEDGEFETCH_ROBOTS_BYPASS_HOSTS/robots_bypass_hosts(on worker nodes); every bypass call is logged inhub_events. robots.txtandContent-Signalblocks are never handed to a fallback service.
Markdown negotiation and HTTP 402
Section titled “Markdown negotiation and HTTP 402”- Markdown for Agents: Page scrapes negotiate
Accept: text/markdown, text/html;q=0.9, */*;q=0.8first, saving bandwidth on origins that serve Markdown natively (contentSource: "origin-markdown"). - HTTP 402 / x402: When a site answers with
402 Payment Required(includingx402payment headers or JSON terms), EdgeFetch classifies the response asoutcome: "payment_required", reports the price and terms to the calling agent, and never pays automatically.
Rate limits (“3 strikes and you’re out”)
Section titled “Rate limits (“3 strikes and you’re out”)”- On HTTP
429(when not an anti-bot challenge), EdgeFetch performs up to 3 adaptive attempts, honouring the origin’sRetry-Afterheader (or exponential4s→10sbackoff with jitter whenRetry-Afteris absent) within the call’s remaining deadline before returningrate_limitedand placing that host/IP into a fleet cooldown. - Each host has its own concurrency limit and pacing (
>= 2son store catalogs), across the whole fleet.
Challenges and CAPTCHAs
Section titled “Challenges and CAPTCHAs”EdgeFetch recognises challenge pages from Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF,
Imperva and others. When it sees one, it stops and returns blocked: true. It ships no
stealth plugins, rotating proxies or CAPTCHA solvers.
Identifying itself
Section titled “Identifying itself”Requests carry an EdgeFetch/<version> User-Agent. Operators can sign requests with RFC 9421
HTTP Message Signatures (Ed25519) and publish their key at
/.well-known/http-message-signatures-directory on their hub, so a site can check who sent the
traffic. See For website owners.
