AI crawlers in your server logs

What it does

Your server's access log records every request AI crawlers make to your site. Upload it on a project's AI crawlers page (under AI visibility) and Asheard shows which AI crawlers read the site, what they fetch, what fails, and whether the pages they read get cited in AI answers and clicked on Google. It costs no credits.

• Per crawler: its requests and pages over the latest 30 days of your logs, the share that failed, when it was last seen, and whether your robots.txt lets it in.
• Requests per day, split into AI search crawlers, live answers (a page an assistant opens while answering someone) and training crawlers.
• Per page, the whole path: crawled (these logs), read and cited (in the AI answers we track for the project, last 30 days) and clicked (Google Search, when Search Console is connected; visits from AI assistants, when Google Analytics is connected).
• What to fix, most urgent first: see below.

The crawlers counted

AI search crawlers build the index an AI search answers from: OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot and MistralAI-Index. Live-answer crawlers open a page while an assistant answers someone: ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User. Training crawlers collect pages to train models: GPTBot, ClaudeBot, MistralAI-Training, meta-externalagent, Bytespider and CCBot.

Google's AI Overviews and AI Mode and Microsoft's Copilot use the search crawlers Googlebot and Bingbot, so their AI features can't be told apart from search in a log, and they aren't counted here. Google-Extended and Applebot-Extended are robots.txt settings, not crawlers: they never show up in a log.

Uploading logs

Managers and admins upload and delete logs; everyone who can see the project can see the page. Choose the site the log belongs to (one of the project's domains or the brand's website) and upload up to 5 files of up to 10 MB each, plain or compressed (.gz).

• The format is the combined log format, the default of Apache and nginx, also with the site name in front (Apache's vhost_combined). In a log with several sites, only the lines of the chosen site count.
• On your server, the logs are usually in /var/log/nginx/ or /var/log/apache2/ (or /var/log/httpd/). Hosting control panels often offer them as raw access logs.
• Logs bigger than 10 MB: keep only the AI crawler lines first. The upload form shows a one-line command that does it; AI crawlers are usually a small share of a log.
• Reading a log takes a minute; reload the page to see it. One upload is read at a time per project.
• Uploading days that are already there replaces them for that site, so the same log can be uploaded again without counting twice.

What Asheard keeps

Only the AI crawlers' requests: for each day, crawler, page and response status, how many requests there were. Visitors' IP addresses, other requests, user agents and referrers are never stored. The uploaded files are deleted as soon as they're read, also when reading fails. Deleting an upload removes what it counted. Crawl data is kept for about 13 months back from the newest day.

Fake AI crawlers

Anyone can put a crawler's name in a request. The companies behind ChatGPT, Claude, Perplexity and Mistral publish the addresses their crawlers use, so each request that names one of their crawlers is checked against that list while the log is read: requests from other addresses are fake, counted separately and left out of every other number. Crawlers without a published list are counted as they come.

If your site is behind a proxy or CDN that the server logs instead of the visitor, no request matches, so nothing can be checked: the page says so and every request that names a crawler counts. To have them checked, set the server to log the visitor's address that the proxy or CDN passes on.

What to fix

• robots.txt blocks an AI search or live-answer crawler, for the whole site or for some of your most crawled pages. Blocking training crawlers is your choice, so it isn't flagged. The check reads your live robots.txt.
• A crawler's requests get refused (401, 403 or 429) for 10% or more of its requests: usually a firewall, a CDN's bot protection or a rate limit.
• 10% or more of AI crawler requests hit missing pages (404 or 410): redirect them to the right pages.
• AI crawler requests get server errors (5xx).
• Pages crawled a lot but never cited in the AI answers we track, each with a link to audit the page.
• Requests from fake AI crawlers, for information.

Good to know

• Days are the dates in your log, in your server's time zone.
• ChatGPT-User and Perplexity-User don't follow robots.txt, by their vendors' own account: they open the pages a person asked about. The crawler table marks them.
• Read and cited count the AI answers we track for the brand's website and its subdomains, so they show for that site only.