sjdirect/commoner
Common libs, helpers and utilities
Discovered public repositories for sjdirect in the GitHub catalog.
Common libs, helpers and utilities
The Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. This project provides an easy-to-use class, implemented in C#, to work with robots.txt files.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.