Robots.txt Generator: Agent Groups, Rules and Sitemaps
Build crawler agent groups, allow/disallow paths and sitemap directives locally with newline-injection checks and complete text export.
robots-txt-generatorLOCAL WORKSPACE
Enabling local controls…
Generator settings
Maximum 30 groups, 100 rules per group and 30 sitemaps. Paths start with /; percent-encode spaces and #. robots.txt is public guidance, not access control or an indexing guarantee.
Inputs stay in page memory without transmission, storage or URL insertion.
Static source date: 2026-10-17
Preview and complete output
Set fields or load a sample, then generate.
How to use this tool
Add groups and rules, provide optional HTTP(S) sitemap URLs, generate and review before publishing at your site's /robots.txt.
When not to use it
robots.txt is public crawler guidance, not access control or an indexing guarantee. Paths with spaces or # require percent encoding.
Read the guide →