Scraper GitHub Repositories (2026) · Open Source Tools

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

Free open-source desktop app that scrapes JAV metadata and generates NFO + cover art for Jellyfin, Emby & Kodi. No Docker, no CLI — one-click install on Windows & macOS. 8 built-in sources + optional Metatube federation (30+ providers), actress collections, cross-language tag aliases, and a REST API for AI agents.

Create agents that monitor and act on your behalf. Your agents are standing by!

NewPipe's core library for extracting data from streaming sites

An ergonomic, privacy-aware Python HTTP Client

Client-side downloader app built with Capacitor and Tauri.

Unofficial MyAnimeList PHP+REST API which provides functions other than the official API

Indonesia Stock Exchange (Bursa Efek Indonesia) Scraper

AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.

Turn any website into a structured API. Extract, automate, search and monitor the web.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

🔮 A Node.js scraper for humans.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.

Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people and characters

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.