papriwalprateek / domain-crawler

This is a simple domain crawler built in ruby. The primary purpose of this crawler is to get store all the valid urls(a tags) of any domain(website). Furthurmore, it tries to avoid certain redundant webpages which does not contain important content.