← All tools

robots.txt Tester: Matched Rules & Crawl Decisions

Test pasted robots.txt rules for GPTBot, ClaudeBot, Googlebot or a custom crawler. Inspect matching lines and precedence locally.

robots-txt-testerLOCAL WORKSPACE

Enabling local controls…

1. Supply robots.txt

Analyzes supplied text as the URL origin’s robots.txt; never fetches a site. Limits: 1 MiB UTF-8, 10,000 lines, 5,000 rules, 2,048 path characters and 32 stars per rule, 2,000,000 matching steps. Excess input is rejected as a whole.

Use these tokens to test a draft ruleset. Consult each provider for actual behavior and exceptions.

Paths follow RFC 9309. Group selection uses a case-insensitive substring match for product tokens and combines the most specific groups. Case-sensitive paths, longest match, Allow wins ties. Supports * and trailing $, matching path plus query without fragment; normalizes UTF-8 and percent octets. Crawl-delay is shown as an advisory extension. Other unsupported directives produce warnings. Does not simulate Google’s 500 KiB fetched-file truncation. Crawl permission is not indexing or access control.

RFC 9309 · Google rules · OpenAI crawlers · Google-Extended

2. Matching rule and reason

Paste rules and a URL or path, then test.

How to use this tool

Paste robots.txt, specify the crawler and URL, then inspect the selected group, matching rules and winning line. Read the supported matching dialect before deploying.

When not to use it

Allowing crawling is not a promise of indexing. robots.txt is public guidance, not access control; pasted text cannot prove the file's live location, HTTP status or crawler behavior.

Read the guide →