onewaterdrop/node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
A scalable web crawler framework.
This repository is cataloged as part of our automated global GitHub synchronization. Full telemetry, velocity snapshots, and code summaries are scheduled for continuous enrichment.
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.
a web crawler