NodeJS robots.txt parser with support for wildcard (*) matching.

A pure-Python robots.txt parser with support for modern conventions.

An extensible robots.txt parser and client library, with full support for every directive and specification.

Alternative robots parser module for Python

robots.txt parser for Rust.

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Go robots.txt parser

Robots.txt parser and fetcher for Elixir

A lightweight and simple robots.txt parser in node

Chrome extension that audits which paths a site disallows in robots.txt and how many of them Google has indexed anyways.

Visual App for Testing URLs and User-agents blocked by robots.txt Files

🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library

A parser for robots.txt with support for wildcards. See also RFC 9309.

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

Robots.txt parser and generator - Work in progress

💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.

A scalable job indexing system that collects job metadata from career pages (Indian & Global Tech Companies) and exposes them via a centralized REST API.

Parse robots.txt and traverse sitemaps.

Parse robots.txt and sitemaps using dotnet