onewaterdrop/webmagic
A scalable web crawler framework.
Public GitHub Catalog
Discovered Oct 2, 2026
onewaterdrop / Crawler-1
Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.
About this discovery
This repository is cataloged as part of our automated global GitHub synchronization. Full telemetry, velocity snapshots, and code summaries are scheduled for continuous enrichment.
#25309966GitHub System ID
onewaterdropOrganization / User
PublicVisibility
ActiveCatalog Status
More from onewaterdrop
onewaterdrop/node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
onewaterdrop/distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
onewaterdrop/crawler
a web crawler