Robots.txt Generator

Common robots.txt patterns

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Wildcards (Google + Bing)

• * matches zero or more chars — Disallow: /*? blocks any URL with a query string
• $ matches end of URL — Disallow: /*.pdf$ blocks PDFs
• Allow: overrides Disallow for more specific paths

Don't block your CSS/JS

Google has explicitly warned against blocking /assets/ or /static/ — modern crawlers render your page like a browser. Without CSS/JS they may flag your site as broken or non-mobile-friendly.

Privacy

The robots.txt is built locally in your browser. Nothing is uploaded.