jgrietveld/Crawler
Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.
Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser with XPath and CSS selector support.
Public repository record indexed from GitHub. Explore verified star velocity metrics, source code repositories, and curated developer tool directories across the GitHubRepo ecosystem.
Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.