0B5C/Arachne
Arachne is a simple in python written document crawler. At the moment Arachne only supports text files (*.txt), but this will be extended. I plan to integrate a pluginsystem for different crawling methods (web, irc, pdf, ..). Take a look at the search engine ("Horrorscope") ;)