What is… · updated 2026-08-17

What is GPTBot, and why is it getting a 403 on my site?

GPTBot is OpenAI's web crawler — the bot that reads public pages so ChatGPT can describe them. If GPTBot receives a 403 on your site, the block is usually in your CDN or firewall rather than your robots.txt, which is why editing robots.txt so often changes nothing.

1

Why is GPTBot getting a 403 when my robots.txt allows it?

Because robots.txt and your CDN are two different gates, and the CDN is the one that answers first.

robots.txt is a stated policy. It is a text file asking well-behaved bots to follow certain rules, and it has no power to enforce anything. Your CDN or firewall sits physically in front of your server and can refuse a request outright, before your site is ever consulted.

So when Cloudflare's AI crawler blocking is switched on, GPTBot receives a 403 from Cloudflare. Your robots.txt is never read, your server never sees the request, and your analytics show nothing at all.

2

How do I tell where the block is coming from?

Compare two requests to the same page: one as a normal browser, one as GPTBot. What comes back tells you which layer is responsible.

  • Browser gets 200, GPTBot gets 403 — something in front of your site is blocking by user-agent. Check your CDN first.
  • Browser gets 200, GPTBot gets 402 — pay-per-crawl is switched on. That is a deliberate monetisation setting, not a misconfiguration.
  • Both get 200, but GPTBot gets a 'Just a moment…' page — a bot challenge. The crawler cannot solve it, so it leaves with nothing.
  • Both get 200 with the real page — GPTBot is not blocked. If you are still invisible in ChatGPT, the problem is your content, not your access.
3

How do I allow GPTBot?

On Cloudflare, open your domain and go to AI Crawl Control (older dashboards call it Security → Bots → 'AI Scrapers and Crawlers'). Allow the crawler purposes you want — Search and Agent are the ones that put you into ChatGPT's answers. Then check Security → WAF for any custom rule matching AI user-agents, since a hand-written rule will override the managed setting.

On other hosts, look for a 'bot protection', 'bot fight mode' or 'AI crawler' toggle in the security settings.

Then confirm it actually worked by requesting your page as GPTBot again. Changing the setting and assuming is how sites end up blocked for another six months.

4

Common questions

Does a 403 for GPTBot mean ChatGPT can't see my site at all?

It means ChatGPT cannot crawl it. ChatGPT might still know about your business from other sources that mention you — directories, Reddit, news — but it cannot read your own pages, so it cannot quote your pricing, features or wording.

Is GPTBot the same as ChatGPT-User?

No. GPTBot crawls pages in advance. ChatGPT-User fetches a page live when a person asks about it during a conversation. OAI-SearchBot is a third one, powering ChatGPT's search results. They can be allowed or blocked independently.

I allowed GPTBot but ChatGPT still doesn't mention me. Why?

Access is necessary but not sufficient. Once a crawler can read you, whether you get mentioned depends on whether your pages actually answer the questions people ask, and whether other sources corroborate that your business exists.

Check this on your own site

Get a free Launch Score for your site in under a minute — no signup. The full report tests whether each AI crawler can actually reach you, and tells you exactly what to change if it can't.

Scan my site free →

Related: What is AI crawler? · What is AEO? · What is llms.txt?

What is GPTBot, and why is it getting a 403 on my site? · Scanvify