---
title: "How to see which AI crawlers visit your website - Tracwell"
description: "AI crawlers like GPTBot, ClaudeBot and PerplexityBot never run your analytics script. Learn how to see them, what each one is for, and how to control them."
canonical: "https://tracwell.app/guides/track-ai-crawlers"
markdown: "https://tracwell.app/guides/track-ai-crawlers.md"
---

[Guides](https://tracwell.app/guides)

# How to see which AI crawlers visit your website

Updated October 15, 2026

Short answer

AI crawlers do not run JavaScript, so browser analytics never records them. To see them, record requests on your server or at the edge and match the crawler’s user agent, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot. Group them by purpose (training, search indexing, or fetching a page to answer someone) because each means something different for your site.

## Why your analytics shows no AI crawlers

Browser analytics records a visit when the page runs a tracking script. Crawlers request the HTML and leave; most never execute JavaScript. Your analytics can show no AI activity at all while your server answers thousands of crawler requests a week.

## The three kinds of AI crawler

- Training crawlers collect pages to train models, for example GPTBot (OpenAI), ClaudeBot (Anthropic) and AI2Bot (Ai2).
- Search crawlers build the index an assistant searches, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Answer fetchers load a page because a person asked about it, for example ChatGPT-User, Claude-User and Perplexity-User. These are the closest thing to a real reader.

Some crawlers, such as Amazonbot, Applebot, Bytespider and CCBot, serve several purposes or feed public datasets, so their requests do not point to a single use.

## Ways to see crawler requests

- Server logs record every request with its user agent, but you filter and group them yourself.
- Your CDN, such as Cloudflare, shows bot traffic in its own dashboard, separate from your product analytics.
- Server-side tracking sends each matching request to your analytics, so crawler activity sits next to your traffic and conversions.

## Track AI crawlers in Tracwell

1. Generate a server key in Settings → Website → Server keys and store it as a server-only secret.
2. Install the tracwell package and add one trackAiCrawler call to your middleware, proxy or Worker. The setup guide covers Next.js, Cloudflare Workers, Astro, SvelteKit and more.
3. Open Overview → AI crawlers to see requests by provider, crawler, purpose and page.

Crawler tracking runs on your server next to your existing browser script and adds nothing to the page.

## What crawler numbers do and do not mean

- User agents are self-declared. A request that says GPTBot is very likely OpenAI, but the header alone does not prove it.
- Requests are not people. An answer fetch means an assistant read the page for someone; it does not tell you who, or whether they clicked through.
- To see whether assistants send real visitors, pair crawler data with AI referrals.

## Control which crawlers can read your site

Use robots.txt to allow or block crawlers by name. Blocking a training crawler such as GPTBot does not block the same provider’s search crawler, so you can opt out of training while staying visible in AI search. Robots rules are requests, not enforcement; use your CDN’s bot controls if you need to block traffic outright.

## Frequently asked questions

**Does blocking GPTBot remove my site from ChatGPT search?**

No. OpenAI uses OAI-SearchBot for search and ChatGPT-User for fetches a person asks for. Blocking GPTBot only signals that your content should not be used for training.

**Why does Google not appear as an AI crawler?**

Google uses its regular Googlebot for search, including AI Overviews. Google-Extended is a robots.txt control token rather than a separate crawler, so it never appears in request logs.

**Do crawler requests count as visitors?**

No. Tracwell reports crawler requests separately from visitors and page views, so they do not inflate your traffic numbers.

## Keep reading

- [AI crawler setup guide](https://tracwell.app/docs/ai-crawlers)
- [Track traffic from ChatGPT](https://tracwell.app/guides/track-chatgpt-traffic)