Robots.txt Generator
Build a correct robots.txt in seconds: block private folders, control individual crawlers including AI bots, and point search engines to your sitemap.
Default rules
Individual crawlers
Override the default for specific bots. AI crawlers collect content to train AI models.
robots.txt
Upload the file to the root of your domain so it opens at https://yourdomain.com/robots.txt.
Unlock All 16 Free SEO Tools
- Keyword research, SEO audit & rank checkers
- Unlimited use, 100% free
- No password to remember
- Stay signed in for 1 year
How to use the Robots.txt Generator
- 1
Choose whether search engines are allowed by default.
- 2
Add folders to block and set rules for individual crawlers, including AI bots.
- 3
Download robots.txt and upload it to the root of your domain.
Frequently asked questions
What is robots.txt?
robots.txt is a text file at the root of a website that tells search engine and AI crawlers which parts of the site they may or may not crawl.
Can I block AI crawlers like GPTBot?
Yes. Add a User-agent rule for each AI crawler, such as GPTBot, ClaudeBot or Google-Extended, with Disallow: /.
Does robots.txt hide a page from Google?
Not completely. It stops crawling, but a blocked page can still appear if other sites link to it. Use a noindex meta tag to keep a page out of search results.
Want Experts to
Fix It For You?
Our SEO and development team can fix the issues these tools find and grow your traffic. Ask for a free consultation.
- Fast Response
- No Obligation
- 100% Confidential