Built something? We create video reels & spotlights for GitHub projects.Promote your project →
Catalog / vekerdyb / simplerobotparser
Public GitHub Catalog Discovered Sep 26, 2026

vekerdyb / simplerobotparser

A python script to fetch and parse robots.txt files for web-crawlers

View repository on GitHub ↗ View creator profile Browse directory

About this discovery

This repository is cataloged as part of our automated global GitHub synchronization. Full telemetry, velocity snapshots, and code summaries are scheduled for continuous enrichment.

#5996332GitHub System ID
vekerdybOrganization / User
PublicVisibility
ActiveCatalog Status

More from vekerdyb

↗

vekerdyb/silly-search

A simplistic search engine demonstrating how easy it is to python.

Discovered
FOR MAINTAINERS

Built something? Put it in front of millions of developers.

We make a short reel about your project and post it across YouTube, Instagram, Threads, and X. Send a link, we do the rest.