🤖 Free tool

Online Robots.txt Tester

Test whether your robots.txt lets AI crawlers like GPTBot, ClaudeBot, and PerplexityBot reach your pages. The Online Robots.txt Tester is 100% free — enter your URL, and we send the full report straight to your inbox.

Free · No signup · Report emailed in ~1 minute

What is the Online Robots.txt Tester?

The Online Robots.txt Tester fetches your robots.txt and checks whether AI crawlers — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended — are allowed, blocked, or unmentioned. It also verifies your sitemap reference and flags rules that quietly hide you from AI search. Results are emailed to you.

What the Online Robots.txt Tester checks

  • Whether robots.txt exists and parses correctly
  • Access for each major AI bot: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended
  • Traditional crawlers: Googlebot and Bingbot
  • Whether a sitemap.xml is declared and reachable
  • Overly broad Disallow rules that block AI discovery
  • Meta robots and X-Robots-Tag directives that conflict with robots.txt

How it works

  1. 1

    Enter your URL

    Paste any page on your domain — we read robots.txt from the root.

  2. 2

    We test every AI bot

    We evaluate each crawler individually against your rules to see who can reach you.

  3. 3

    Get your report by email

    You receive a per-bot allowed/blocked breakdown and the fixes to unblock AI search.

Frequently asked questions

Which AI crawlers should robots.txt allow?

To be discoverable in AI search, most sites allow GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Google AI). Blocking them removes you from those engines.

Will this change my robots.txt?

No. The tester only reads and evaluates your existing robots.txt. It shows you exactly which rules to change, but never modifies your site.

What happens if I block GPTBot?

If GPTBot is disallowed, OpenAI’s crawler will not index your pages, so ChatGPT and SearchGPT are far less likely to cite or recommend your site.

Is the Online Robots.txt Tester really free?

Yes. You can scan a website at no cost and receive the full report by email. There is nothing to install and no credit card required.

Also known as

The same free tool, by the names people search for it:

Robots.txt TesterRobots.txt CheckerRobots.txt ValidatorRobots.txt AnalyzerRobots.txt AuditRobots.txt AI Crawler TesterAI Crawler TesterAI Bot TesterGPTBot Robots.txt TesterClaudeBot Robots.txt TesterPerplexityBot Robots.txt TesterGoogle-Extended Robots.txt TesterFree Robots.txt TesterInstant Robots.txt TesterFree Robots.txt CheckerFree Robots.txt Validator

More free AI visibility tools

🛰️

AI Visibility Checker

See how ChatGPT, Perplexity, Gemini, and Google AI Overviews find, read, and recommend your website — scored 0–100.

📄

llms.txt Checker

Check whether your site publishes a valid llms.txt file with the sections AI systems expect — and what to add.

🧩

Schema Markup Checker

Validate the JSON-LD structured data on your page — Organization, FAQ, Product, Article, and more.

📖

AI Readability Checker

Measure whether AI models can cleanly extract answers from your content — headings, structure, and clarity.

Fix what the scan finds — automatically

CookMyRank turns every issue this tool surfaces into a deployable fix — schema, metadata, llms.txt, and content — that you approve before it ships.

See how it works