Measuring AI crawler traffic in your server logs

“AI traffic” conflates two different things, and separating them is the first job of any honest measurement. AI crawler traffic is bots fetching your pages — to train models, to build a search index, or to ground a live answer. AI referral traffic is humans clicking through to your site from an assistant’s answer; that one lives in client-side analytics and gets its own guide. This guide covers the crawlers — and the crucial point is that your analytics tool cannot see them. Most AI crawlers do not execute JavaScript, so they never fire a tag. Crawler measurement is a server-log and CDN-log discipline, closer to classic log analysis than to anything in the tag manager.

The crawlers worth knowing

The table below lists the major operators’ user-agent tokens and how each can be verified, as documented in each operator’s own bot documentation — last reviewed 2026-07-11. Operators add, rename, and re-scope crawlers often, so treat any copy of this table older than a quarter as stale and re-check the tokens and IP lists against the operator’s published docs before relying on them (the same discipline this guide demands of every published number).

Operator	User-agent token(s)	Purpose	Verification
OpenAI	GPTBot	Model-training crawl	Published IP ranges (openai.com/gptbot.json); rDNS not guaranteed
OpenAI	OAI-SearchBot	Search index	Published IP ranges
OpenAI	ChatGPT-User	Live retrieval for a user prompt	Published IP ranges
Anthropic	ClaudeBot, Claude-User, Claude-SearchBot	Training / user-triggered fetch / search	Published IP ranges (anthropic.com docs)
Google	Google-Extended	Gemini-training opt-out control — a robots.txt token only, not a separate fetcher	n/a — it gates Googlebot
Google	Googlebot	Main index, which also powers AI Overviews	Reverse + forward DNS to googlebot.com
Perplexity	PerplexityBot, Perplexity-User	Index / live fetch	Published IP ranges; documented UA-masking controversy — IP verification is mandatory
Microsoft	bingbot	Bing index, which powers Copilot	rDNS to search.msn.com
Common Crawl	CCBot	Open crawl used by many model trainers	UA plus IP heuristics
Apple	Applebot, Applebot-Extended	Apple Intelligence	rDNS to applebot.apple.com
Amazon	Amazonbot	Alexa / model training	rDNS to crawl.amazonbot.amazon
Meta	meta-externalagent, FacebookBot	Llama training / link previews	Published ranges
ByteDance	Bytespider	TikTok / Doubao training	Aggressive crawler; UA plus IP heuristics

Note the three distinct purposes hiding in one column: training crawls (GPTBot, ClaudeBot, Bytespider), search-index crawls (OAI-SearchBot, PerplexityBot), and live retrieval (ChatGPT-User, Claude-User, Perplexity-User) — a fetch made because a human just asked a question your page might answer. For anyone measuring AI visibility, those are three very different signals, and lumping them into one “AI bots” number destroys the information.

The trap: user agents are free to fake

A user-agent string is a self-declared HTTP header. Anyone can send GPTBot. Counting log lines by UA token therefore fails in both directions: scrapers spoof well-known AI crawler UAs to look benign and slip through allowlists, and some operators have been documented fetching with browser-like UAs that no token search will catch. A UA-only count is not a measurement — it is an upper bound on politeness. Any number you publish or act on should be verified against the request’s source, by at least one of the following methods.

1. Published IP ranges. Most major operators now publish their crawler egress IPs as JSON (OpenAI’s gptbot.json is the pattern; Anthropic, Perplexity, and others publish equivalents as of mid-2026). Match the connecting IP against the CIDR list: exact and fast, but the lists change, so fetch them on a schedule and record which version of the list verified each hit.

2. Reverse-plus-forward DNS. For Googlebot, bingbot, Applebot, and Amazonbot, the operator’s documented method is rDNS: resolve the IP to a hostname, check it ends in the operator’s domain, then forward-resolve that hostname and confirm it returns the same IP. The forward step is what defeats spoofing — anyone can put lies in their own reverse zone, but not in googlebot.com’s forward zone:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1   # matches → verified

3. TLS fingerprinting (JA3/JA4). A TLS client’s ClientHello — cipher suites, extensions, ordering — fingerprints the software making the request. A request claiming to be GPTBot but presenting the JA4 fingerprint of headless Chrome or python-requests is lying, whatever its IP story. Fingerprinting requires visibility at the TLS terminator (CDN, load balancer, or an eBPF-based capture on your own edge), which is why it is usually the last method teams add — and the strongest.

Instrumenting it in practice

• Behind a CDN (fastest path): Cloudflare and its peers already classify verified bots and expose bot-category and JA4 fields in their request logs. Push the logs to object storage and query them with whatever engine you like — the crawler table above becomes a simple join.
• Raw nginx/Apache logs: a small scheduled job — DuckDB or Polars is plenty — parses the access log, joins the source IP against the cached operator IP lists (expand CIDRs to integer ranges and join on ip_int between range_start and range_end), applies rDNS checks for the DNS-verified operators, and emits a daily ai_crawler_hits table with operator, agent, purpose, verified columns. Keep the unverified UA matches in a separate column rather than discarding them: the gap between claimed and verified is itself a spoofing metric.
• Zero-instrumentation capture: eBPF-based tools can observe HTTP server traffic without touching application code, which suits infrastructures where log access is awkward — the same event stream then feeds the pipeline like any other log source.

Whatever the plumbing, report at minimum: verified hits per operator per day, the pages fetched (training crawls concentrate on your most-linked content; live retrieval concentrates on whatever users are asking about right now), and the verified-to-claimed ratio.

robots.txt: a request, not a control

The blocking surface is robots.txt, and it is worth stating plainly what it is: a voluntary convention. Reputable operators honor it; nothing forces anyone to.

# Block model-training crawls
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Opt out of Gemini model training while remaining in Google Search
User-agent: Google-Extended
Disallow: /

Per-operator nuances matter. Google-Extended is not a crawler — it is a token that tells Googlebot not to use your content for Gemini training, without touching your Search presence. Blocking GPTBot stops OpenAI’s training crawl but is a separate decision from blocking OAI-SearchBot (search visibility) or ChatGPT-User (live answers that may cite you) — many sites want to block training while staying citable, and the tokens let you split that decision. But if a crawler ignores the file, robots.txt does nothing; actual enforcement means WAF or edge rules keyed to the verified identity signals above, which is precisely why verification and policy belong in the same pipeline. Measure first, then decide, then — if needed — enforce.

Across the wider web, robots.txt posture toward AI crawlers is itself a measurable, moving statistic; it is the kind of dimension the State of Web Tracking report exists to track over time. On your own site, the honest baseline is simpler: verified crawler hits by operator and purpose, refreshed IP lists, a dated reference table — and no number in a deck that started life as an unverified user-agent count.