PuerkitoBio/fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Discovered public repositories for PuerkitoBio in the GitHub catalog.
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
tiny Go library to normalize URLs
Polite, slim and concurrent web crawler.
A little like that j-thing, only in Go.