AI Crawler Checker

Check robots.txt rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more, then test your server with AI user agents to catch firewall blocks.

Leave empty to test the home page (/).

How to use the AI Crawler Checker

  1. 1

    Enter your domain (and optionally a path such as /blog/).

  2. 2

    Click Check AI crawlers.

  3. 3

    Allow the AI search bots you want to be cited by, and decide your policy for training bots.

Need expert help?SEO & Digital Marketing Services

Frequently asked questions

What is the difference between training and search bots?

Training bots (GPTBot, ClaudeBot, CCBot) collect pages to train models. Search and user bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch pages to answer and cite them live.

Does blocking Google-Extended affect Google Search?

No. Google-Extended only controls use in Gemini models. Google Search and AI Overviews use Googlebot.

Why is a bot blocked when robots.txt allows it?

A firewall or CDN rule (for example Cloudflare's AI bot blocking) can block bots before robots.txt is read. The live test shows this.

Want Experts to
Fix It For You?

Our SEO and development team can fix the issues these tools find and grow your traffic. Ask for a free consultation.

  • Fast Response
  • No Obligation
  • 100% Confidential
We reply within 24 hours. Your details stay private.
More Free Tools

Keep Optimizing

Citability Scorer

See how easy your page is for AI engines to quote.

First-40-Words Rewriter

Rewrite your intro so it answers in the first 40 words.

Prompt Alignment Checker

Check whether a page answers the prompts users ask AI.

llms.txt Builder

Generate an llms.txt file from your sitemap in seconds.

Entity Clarity Checker

Check if search engines and AI know who your brand is.
View All Tools