EA Digital Solutions
SEOSchemaMeta TagsBriefGEOOG PreviewImageColorsBGUTMRedirectsPHPE-mailSpeedHeadersAI crawlersMockup

robots.txt · llms.txt · GPTBot · ClaudeBot · PerplexityBot

AI crawler check: may ChatGPT, Claude and Perplexity read your website?

Your robots.txt decides which AI crawlers may fetch your pages. This free check evaluates it for 19 crawlers and tokens, separates AI search from AI training and gives you a ready-made robots.txt.

The check fetches exactly two files of your domain: /robots.txt and /llms.txt. No other pages are requested. It evaluates whether the crawlers may fetch the start page (/).

Knowledge

AI crawlers and robots.txt: search, training and assistants

Whether AI search such as ChatGPT, Claude or Perplexity can use your website as a source is first decided by your robots.txt. The vendors separate crawlers for training (e.g. GPTBot, ClaudeBot) from crawlers for search (e.g. OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking only the training crawlers keeps you visible in AI search.

Three kinds of AI crawlers

AI training

GPTBot, ClaudeBot, CCBot …

Collect content for model training. Blocking them is a legitimate decision and does not cost visibility in AI answers.

AI search

OAI-SearchBot, Claude-SearchBot, PerplexityBot …

Build the index AI answers draw their sources from. Blocking them means: not cited there.

Assistants

ChatGPT-User, Claude-User, Perplexity-User …

Fetch a single page live when a person asks about it. Not automatic crawling.

How crawlers read a robots.txt (RFC 9309)

  1. 1.A group starts with one or more “User-agent” lines; the Allow and Disallow lines after it belong to it.
  2. 2.A crawler looks for the group with its product token (case does not matter). Several such groups are merged. Only if there is none does the “*” group apply.
  3. 3.Within the group the longest matching path wins; if Allow and Disallow are equally long, Allow wins. “*” stands for any characters, “$” for the end of the path.
  4. 4.If /robots.txt answers with 4xx (e.g. 404), crawlers may fetch everything. If it answers with 5xx or not at all, they must assume a complete block.

A common trap: as soon as a crawler has its own group, it ignores everything under “*”. If you add “User-agent: GPTBot” with a single rule, your other exclusions no longer apply to GPTBot.

Rather have it done? Book a free call →

AI crawler check: frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses separate crawlers: GPTBot collects content for training, OAI-SearchBot indexes pages for search in ChatGPT. OpenAI’s own example shows allowing OAI-SearchBot while disallowing GPTBot. Anthropic separates ClaudeBot (training) from Claude-SearchBot (search) in the same way.

Does Google-Extended block my site from Google Search?

No. Google-Extended is not a crawler but a robots.txt token. It controls whether content crawled by Google may be used to train and ground Gemini models. According to Google it does not affect inclusion or ranking in Google Search. AI Overviews are part of Search and are not controlled by Google-Extended.

Does every crawler obey robots.txt?

No. RFC 9309 states that the rules are not a form of access authorisation. OpenAI and Perplexity say that fetches triggered by a user (ChatGPT-User, Perplexity-User) may not follow robots.txt. Anyone who really has to keep a crawler out needs rules on the server or firewall.

Do I need an llms.txt?

It is not required. llms.txt is a proposal published on llmstxt.org, not a standard. It summarises a website in Markdown for language models; whether a given AI service reads it is up to that service. It does not replace robots.txt or a sitemap.

How quickly does a change to robots.txt take effect?

RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours. OpenAI states that changes for search take about 24 hours. Until then, an older version may still apply.

What exactly does the check do?

It sends one GET request each to /robots.txt and /llms.txt of the entered domain, follows up to five redirects and parses the robots.txt by the rules of RFC 9309: matching group by product token, otherwise the “*” group, longest matching rule wins, Allow wins a tie. It evaluates the start page “/”. No other page is requested.

Sources

The check shows what your robots.txt says. Whether and how a vendor uses your content is up to that vendor; the result is a technical hint, not legal advice.