Top WEB-CRAWLER GitHub Repositories & Tools (2026)

The web data API to search, scrape, and interact at scale. 🔥

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

High-performance web crawling engine with bindings for 11 languages

Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)