Robots.txt Checker

Read your robots.txt like Google does: groups, sitemaps, syntax mistakes, blocked CSS and JS, "Disallow: /" accidents and crawl-delay. Test any path for Googlebot, Bingbot, GPTBot, ClaudeBot and more.

We read /robots.txt on this domain and check the CSS and JS used on your home page.
Paste a draft to test it before you publish. Leave empty to test the live file.

How to use the Robots.txt Checker

  1. 1

    Enter your domain and, if you like, the paths you want to test.

  2. 2

    Click Check robots.txt.

  3. 3

    Fix blocked resources and risky rules, test your draft here, then publish it.

Need expert help?SEO & Digital Marketing Services

Frequently asked questions

Does robots.txt stop a page from being indexed?

No. It stops crawling. A blocked page can still be indexed from links, without its content. Use a noindex tag to keep a page out of Google.

Which rule wins when Allow and Disallow both match?

Google uses the most specific (longest) matching rule. If both are the same length, Allow wins.

Should I block AI bots like GPTBot?

It is your choice. Blocking GPTBot or ClaudeBot stops AI training crawls; their search bots (OAI-SearchBot, Claude-SearchBot) decide whether you can be cited in AI answers.

Want Experts to
Fix It For You?

Our SEO and development team can fix the issues these tools find and grow your traffic. Ask for a free consultation.

  • Fast Response
  • No Obligation
  • 100% Confidential
We reply within 24 hours. Your details stay private.
More Free Tools

Keep Optimizing

Sitemap Generator

Create XML sitemaps instantly.

Robots.txt Generator

Generate robots.txt files easily.

Index Coverage Reader

Explain every Search Console "not indexed" reason.

Sitemap Auditor

Validate your XML sitemap and test its URLs.

Crawl Budget Analyser

Find URLs that waste Googlebot's crawl time.
View All Tools