The web, fetched politely, on hardware you own.
EdgeFetch is a self-hosted scraping and search server for AI agents. One small Go binary turns pages into clean Markdown over MCP. Your home nodes dial out, so no ports are opened, and nobody bills you per page.
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latestMIT licensed. Signed releases. Respects robots.txt on every hop.

One request, start to finish
Your agent asks for a page. Before anything leaves your network, EdgeFetch checks whether it’s allowed and where it’s going. Every fetch records the same timing fields, and you can send them to any OpenTelemetry collector.
edgefetch scrape https://go.dev/doc/effective_go- robots.txtpath allowed for EdgeFetch
- dial checkresolved IP is public, connect allowed
- dnsresolved12 ms
- connectTCP to port 4439 ms
- tlsTLS 1.3, HTTP/221 ms
- first byte200 OK64 ms
- markdownnavigation and footer removed, code blocks kept
Polite by default, and you can’t turn that off
Site owners see a clear user agent and get their rules respected. These aren’t settings. They’re how the fetcher is built.
- It reads robots.txt first, every time
- Checked before the first request and again on every redirect hop, using the RFC 9309 rules.
- It stops at CAPTCHAs
- Cloudflare, Akamai, DataDome and other challenge pages end the fetch with blocked: true. No stealth plugins, no solvers.
- It backs off when asked
- A 429 or 503 with Retry-After puts that host on cooldown across your whole fleet.
- It never touches your LAN
- Resolved IPs are checked inside the dialer, right before connect, so DNS rebinding can’t reach your router or Home Assistant.
Nothing at home faces the internet
A hub on Cloudflare Workers, a small VPS or one container takes requests from your agents. Worker nodes on a Raspberry Pi, NAS or Home Assistant box connect out to it over TLS WebSockets, so they work behind CGNAT without port forwarding.
Check the release yourself
Every release ships SHA-256 checksums signed with Ed25519 and GitHub SLSA build provenance for each archive and container image. CI runs go vet,staticcheck, govulncheck, the race detector and fuzz tests on every push. The Go worker has one external dependency.
# signature over the checksums
openssl pkeyutl -verify -pubin -inkey release-signing-pub.pem \
-rawin -in checksums.txt -sigfile checksums.txt.sig
# provenance from the GitHub Actions build
gh attestation verify edgefetch_<version>_linux_amd64.tar.gz \
--repo bartrumb/edgefetchPick where to start
- Use it from your editorRun it locally over stdio for Claude Code, Claude Desktop or Cursor.
- Run a hubOne-click deploys to Cloudflare, Render, Railway, Deno or Home Assistant.
- Browse the adaptersJob boards, package registries and open data, with no API keys.
- Saw us in your logs?What EdgeFetch is, and how to block or contact it.
