Skip to content

For website owners

If you saw EdgeFetch in your logs, here’s what it is.

EdgeFetch is open-source software that people run themselves to let their AI assistant read a web page when they ask it to. It is not a search-engine crawler, and it isn’t used to train AI models. Because anyone can run it, traffic comes from many operators, not from one company.

  • Asks for Accept: text/markdown first so sites using Cloudflare Markdown for Agents can serve lightweight Markdown instead of full HTML.
  • Reads and obeys your robots.txt and Content-Signal: ai-input=no directives, including on every redirect.
  • Honours Retry-After on 429 and 503 responses, and limits how fast it hits each host.
  • Treats 402 Payment Required (x402) as a price quote reported back to the user, never an error to retry.
  • Stops when it sees a CAPTCHA or bot challenge. It doesn’t try to get past it.

See Politeness rules for the details.

Add this to your robots.txt:

User-agent: EdgeFetch
Disallow: /

To allow it on part of your site only:

User-agent: EdgeFetch
Allow: /docs/
Disallow: /

Operators can sign requests with RFC 9421 HTTP Message Signatures (Ed25519). A signed request includes Signature, Signature-Input and Signature-Agent headers. Signature-Agent points to the operator’s key directory at /.well-known/http-message-signatures-directory. An unsigned request that only claims to be EdgeFetch could come from anyone.

  • About the software, or to report abusive use of it: bot@edgefetch.dev or a GitHub issue.
  • About traffic from a particular operator: they are encouraged to put their own contact details in the User-Agent, so check the header first.