SEO TOOLS
Robots.txt Generator.
Build robots.txt files with common rules.
Three simple steps
How to use Robots.txt Generator
1
Choose what to allow
Start by allowing all crawlers, then add any paths to disallow.
2
Add your sitemap
Point crawlers at your sitemap.xml URL.
3
Publish at the root
Save the file as /robots.txt at your domain root — it only works there.
Included free
Built for the job
- Free, no sign-up
- Runs entirely in your browser
- No file uploads or storage
- Works on mobile and desktop
Helpful answers
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results with no snippet if it's linked elsewhere. Use a noindex meta tag or header to keep a page out.
Can I have more than one robots.txt?
One per host. example.com and www.example.com are different hosts and each needs its own, as does every subdomain.