A detailed review of server access logs shows that most AI vendors operate several distinct bots rather than a single crawler. Each bot serves a different function, and treating them identically in configuration files leads to unintended consequences.
GPTBot performs scheduled crawls for model training, respects robots.txt, and systematically follows sitemaps. OAI-SearchBot builds the search index used for answers that include citations. ChatGPT-User makes on-demand requests triggered by an individual user query. The same separation exists at other providers: Perplexity runs PerplexityBot for indexing and a separate user agent for live queries, while Anthropic maintains ClaudeBot alongside its own user-triggered agents.
Log analysis reveals three consistent request patterns. Index bots traverse site maps and internal links at regular intervals without correlation to traffic spikes. User agents issue single, targeted requests for specific URLs, often carrying a referrer from the AI service. All bots request robots.txt regularly, request sitemap.xml less frequently, and almost never request llms.txt. Over 515 million recorded AI-bot events, only 408 requests targeted llms.txt.
User-Agent strings are easily forged, so verification relies on reverse DNS or, preferably, comparison against IP ranges published by each vendor. A simple script that extracts IPs from logs and checks them against known OpenAI ranges quickly identifies spoofed traffic from unrelated scrapers.
Because llms.txt receives negligible attention from actual crawlers, site owners achieve better results by ensuring content remains in static HTML, pages load quickly, and robots.txt selectively permits user-agent traffic while blocking training bots. Structured headings and self-contained sections also improve the quality of excerpts returned to users.
