Top WEB-SCRAPER GitHub Repositories & Tools (2026)

The web data API to search, scrape, and interact at scale. 🔥

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)