huanyunz/distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Developer Profile
huanyunz
Discovered public repositories for huanyunz in the GitHub catalog.
huanyunz/crawler
Public repository.
huanyunz/dirbot
Scrapy project to scrape public web directories (educational)