Top WEB-SCRAPING GitHub Repositories & Tools (2026)

The web data API to search, scrape, and interact at scale. 🔥

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

A curated directory of ecommerce intelligence APIs for products, prices, reviews, sellers, and marketplaces.

The only browser automation that bypasses anti-bot systems. AI writes network hooks, clones UIs pixel-perfect via simple chat.

A native Android app that aggregates third-party app stores into one catalogue. Search across all of them at once, compare what each publishes for the same app, download and install APKs from the store you pick, and keep what you installed up to date.

A curated directory of job data APIs and scrapers for listings, hiring signals, salaries, and recruiting.

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

Self-hosted SERP API for AI, SEO & automation. Browser-rendered Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia search with page extraction 🎉

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Provides a list of fresh, working proxy servers (HTTP, HTTPS, SOCKS4 & SOCKS5) with multiple formats available for download.

High-performance web crawling engine with bindings for 11 languages

Proxify is an automated tool that collects and updates fresh SOCKS4, SOCKS5, HTTP/HTTPS proxies and V2Ray configs from public sources. It maintains a regularly updated repository of working proxies, useful for privacy, bypassing restrictions, or network testing.

Free Proxy List Updated every 10 minutes

Cloudflare Turnstile challenge internals: the live request flow, the challenge bundle, and a capture toolkit.

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)