How ChatGPT search finds sites, and how to stay in or out of it

OpenAI runs several user agents, and only one of them decides search visibility. OAI-SearchBot crawls pages so they can be surfaced in ChatGPT's search features. If robots.txt blocks it, the site is not shown in ChatGPT search answers, although it can still turn up as a navigational link.

OpenAI's advice for appearing in results is short: allow OAI-SearchBot in robots.txt, and allow requests from the IP ranges it publishes at openai.com/searchbot.json. A firewall or bot filter that blocks those addresses can undo a correct robots.txt.

GPTBot is the crawler for training OpenAI's generative AI foundation models. Disallowing it signals that a site's content should not be used for training. OpenAI says each setting is independent, so a site can allow OAI-SearchBot to appear in search while disallowing GPTBot.

This is the most common mix-up. Blocking GPTBot to keep content out of training does not, on its own, remove a site from ChatGPT search. Blocking OAI-SearchBot does. When a site allows both, OpenAI may reuse one crawl for both purposes rather than fetching pages twice.

ChatGPT-User fetches a page when a person asks ChatGPT or a Custom GPT a question that needs it. It does not crawl the web automatically and is not used to decide whether content appears in search. Because a user starts the request, OpenAI says robots.txt rules may not apply to it. To manage search opt-outs, OpenAI points site owners to OAI-SearchBot instead.

OAI-AdsBot visits only landing pages submitted as ads on ChatGPT, to check them against OpenAI's policies and judge when an ad is relevant. Its data is not used for model training.

Each crawler identifies itself in its user-agent string, for example OAI-SearchBot/1.4 or GPTBot/1.4, and OpenAI notes the version number may change, so match on the name. When fetching robots.txt, OpenAI may add a robots.txt marker to the user agent, which helps tell those requests apart in logs that omit paths. Each crawler also has its own published IP list.

After changing robots.txt, allow about a day: OpenAI says its search systems can take around 24 hours to adjust. OpenAI's crawler page does not say how ChatGPT search ranks or chooses among the pages it can reach, so crawler access is a requirement, not a promise of being cited.