PuerkitoBio/purell
tiny Go library to normalize URLs
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
This repository is cataloged as part of our automated global GitHub synchronization. Full telemetry, velocity snapshots, and code summaries are scheduled for continuous enrichment.
tiny Go library to normalize URLs
Polite, slim and concurrent web crawler.
A little like that j-thing, only in Go.