GeoLocalSEO › Guides
Is your website blocking ChatGPT? How to check in two minutes
An AI assistant can only describe or recommend a business it is allowed to read. The file that decides this is called robots.txt, and some website builders and security plugins block AI crawlers without telling you.
Step 1: Open your robots.txt
Type your website address followed by /robots.txt into a browser, for example yourbusiness.com/robots.txt. You will see a short text file. If you get an error page, you have no robots.txt, which means nothing is blocked.
Step 2: Look for these names
Each AI company uses named crawlers. These are the common ones. Names change, so check each company's documentation if you need to be exact.
| Crawler name | Who runs it | What it is for |
|---|---|---|
| OAI-SearchBot | OpenAI | Lets your pages appear in ChatGPT search answers |
| GPTBot | OpenAI | Collects pages that may be used to train models |
| ClaudeBot | Anthropic | Collects pages that may be used to train models |
| PerplexityBot | Perplexity | Lets your pages appear in Perplexity answers |
| Google-Extended | Controls use of your pages for Gemini. It does not affect Google Search | |
| Googlebot | Google Search itself |
Step 3: Read the rule under each name
A block looks like this:
User-agent: GPTBot
Disallow: /Disallow: / means "do not read anything on this site." The same rule under User-agent: * blocks every crawler that is not named elsewhere in the file, including search engines.
Step 4: Fix it
To let a crawler in, delete its Disallow: / line or change it to Allow: /. A simple robots.txt that lets everyone read the site looks like this:
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xmlIf your site runs on a website builder, look for a setting named something like "AI crawlers" or "search engine visibility" instead of editing the file by hand.
What this does and does not do
Letting crawlers in makes your site readable. It does not make any assistant recommend you. Nobody can promise that. Readable, accurate pages are the part you control.
Check your own site in 30 seconds
The free Get Found Check reads your home page the way a crawler does and lists every gap with a fix.
Run my free check