robots.txt Tester: Matched Rules & Crawl Decisions
Test pasted crawler rules against a supplied URL and agent, inspect matching lines and understand precedence without crawling a website.
Enabling local controls…
1. Supply robots.txt
Analyzes supplied text as the URL origin’s robots.txt; never fetches a site. Limits: 1 MiB UTF-8, 10,000 lines, 5,000 rules, 2,048 path characters and 32 stars per rule, 2,000,000 matching steps. Excess input is rejected as a whole.
RFC 9309 path semantics with Google grouping: case-insensitive product tokens; merge most-specific groups. Case-sensitive paths, longest match, Allow wins ties. Supports * and trailing $, matching path plus query without fragment; normalizes UTF-8 and percent octets. Unsupported directives are warned. Does not simulate Google’s 500 KiB fetched-file truncation. Crawl permission is not indexing or access control.
2. Matching rule and reason
Paste rules and a URL, then test.
How to use this tool
Paste robots.txt, specify the crawler and URL, then inspect the selected group, matching rules and winning line. Read the supported matching dialect before deploying.
When not to use it
Allowing crawling is not a promise of indexing. robots.txt is public guidance, not access control; pasted text cannot prove the file's live location, HTTP status or crawler behavior.
Read the guide →