lemurproject/clueweb12pp-core
Unifies all code written for processing clueweb12++
Developer Profile
lemurproject
Discovered public repositories for lemurproject in the GitHub catalog.
lemurproject/nabble-tools
Tools for processing our nabble crawl
lemurproject/ygroups-tools
Processing the Yahoo groups crawl (part of clueweb12++)
lemurproject/reddit_crawler
Download the reddit dataset for ClueWeb12++
lemurproject/scrapers
Collection of Scrapers I am writing
lemurproject/warc-tools
Command line utilities for working with WARC files
lemurproject/python-blekko
bindings for the Blekko search engine API
lemurproject/kba-tools
Tools for processing the Trec KBA dataset
lemurproject/nopol
Simple command line tool to export the ClueWeb dataset as HTML files.
lemurproject/oauth-racket
Oauth Implementation in Racket (PLT Scheme)
lemurproject/crawl-browser
Public repository.
lemurproject/warc
Python library for reading and writing warc files
lemurproject/crawling-tools
A collection of tools for running the crawler, including scripts and scrapers to collect seeds.
lemurproject/language-id
language identification module in racket
lemurproject/site-sampling
Code to sample forums and check stats on pages and their information content
lemurproject/heritrix-jobs
Configuration files and scripts needed to run the Heritrix jobs
lemurproject/forum-seed
Code and other stuff for discussion-forum seeds.
lemurproject/clueweb12pp
Extension to the Clueweb12 dataset
lemurproject/heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.