Skip to content

The web, fetched politely, on hardware you own.

EdgeFetch is a self-hosted scraping and search server for AI agents. One small Go binary turns pages into clean Markdown over MCP. Your home nodes dial out, so no ports are opened, and nobody bills you per page.

go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest

MIT licensed. Signed releases. Respects robots.txt on every hop.

The EdgeFetch raven perched on a circuit-board ledge

One request, start to finish

Your agent asks for a page. Before anything leaves your network, EdgeFetch checks whether it’s allowed and where it’s going. Every fetch records the same timing fields, and you can send them to any OpenTelemetry collector.

edgefetch scrape https://go.dev/doc/effective_go
  1. robots.txtpath allowed for EdgeFetch
  2. dial checkresolved IP is public, connect allowed
  3. dnsresolved12 ms
  4. connectTCP to port 4439 ms
  5. tlsTLS 1.3, HTTP/221 ms
  6. first byte200 OK64 ms
  7. markdownnavigation and footer removed, code blocks kept
An example trace. Timings vary with your network.

Polite by default, and you can’t turn that off

Site owners see a clear user agent and get their rules respected. These aren’t settings. They’re how the fetcher is built.

Read the politeness rules

It reads robots.txt first, every time
Checked before the first request and again on every redirect hop, using the RFC 9309 rules.
It stops at CAPTCHAs
Cloudflare, Akamai, DataDome and other challenge pages end the fetch with blocked: true. No stealth plugins, no solvers.
It backs off when asked
A 429 or 503 with Retry-After puts that host on cooldown across your whole fleet.
It never touches your LAN
Resolved IPs are checked inside the dialer, right before connect, so DNS rebinding can’t reach your router or Home Assistant.

Nothing at home faces the internet

A hub on Cloudflare Workers, a small VPS or one container takes requests from your agents. Worker nodes on a Raspberry Pi, NAS or Home Assistant box connect out to it over TLS WebSockets, so they work behind CGNAT without port forwarding.

AI agents connect to the hub over MCP. Three worker nodes at home open outbound WebSocket connections to the hub. No inbound connections reach the home network.Claude Code, Cursoryour agentsHubMCP, queue and cacheMCPyour networkRaspberry PiHome AssistantDocker on a NASoutbound WSS only

How the hub and workers fit together

Check the release yourself

Every release ships SHA-256 checksums signed with Ed25519 and GitHub SLSA build provenance for each archive and container image. CI runs go vet,staticcheck, govulncheck, the race detector and fuzz tests on every push. The Go worker has one external dependency.

Verify a release step by step

# signature over the checksums
openssl pkeyutl -verify -pubin -inkey release-signing-pub.pem \
  -rawin -in checksums.txt -sigfile checksums.txt.sig

# provenance from the GitHub Actions build
gh attestation verify edgefetch_<version>_linux_amd64.tar.gz \
  --repo bartrumb/edgefetch

Pick where to start