Spider GitHub Repositories (2026) · Open Source Tools

Foundational low latency web data collecting in Rust

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server

A tool for download asmr media from asmr.one(Thanks for the asmr.one)

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

ronin-web is a collection of useful web helper methods and commands.

Ruby gem to detect bots and crawlers via the user agent