Which Crawlers Actually Visit a Brand-New Site? We Measured Ours

The short version

What we measured, and how

This site is small and new: the first request in our logs is dated 2026-06-28, and it has a few dozen articles. That makes it a useful specimen — it has no backlink profile, no brand, and nothing to attract crawlers except being on the public web.

We took every line of the nginx access log for this domain between 2026-07-19 and 2026-08-12 — 27,906 requests in total — and classified each one by user agent. Classification uses a fixed list of user-agent tokens applied identically to every line, so the same log always produces the same table; a request counts once per hit. We report requests and distinct URLs separately because a crawler that hits one URL two hundred times is doing something very different from one that walks the whole site.

Two caveats worth stating plainly. User agents are self-reported and can be forged, so these counts describe what identified itself as each crawler, not verified identity. And this is one small site — the ratios here are a data point, not an industry average.

AI crawlers came far more often than Google did

The headline result: crawlers that collect for language models made 1,304 requests — 31% of all identified crawler traffic — while Googlebot made 155, a ratio of about 8.4:1. The AI side is not one bot but many: ClaudeBot (358), PerplexityBot (236), OAI-SearchBot (227), GPTBot (208), meta-externalagent (105).

Google's own crawling was modest in both volume and reach — 155 requests across 70 distinct URLs. For a site of this size that is a light touch.

If you have been assuming that a new site's server load is mostly search engines discovering you, that assumption is out of date. On this site the language-model crawlers are the main audience, and they arrived without being invited.

The busiest crawler was Amazonbot, not a search engine

Amazonbot (Amazon) was the single most active crawler at 623 requests across 408 distinct URLs — more than any search engine in the sample.

This is the part that surprises people who have not looked at their own logs: the crawler budget of a small site is spent on companies that will never send it a visitor. Only a handful of the crawlers below have any path to sending you traffic at all.

Who actually reads sitemap.xml

Sitemaps are supposed to be how you tell crawlers what exists. In practice most crawlers in our sample never asked for ours.

• ClaudeBot — 111 requests for sitemap.xml
• bingbot — 48 requests for sitemap.xml
• GPTBot — 25 requests for sitemap.xml
• AhrefsBot — 6 requests for sitemap.xml
• Googlebot — 2 requests for sitemap.xml

Googlebot requested it 2 times in 25 days. Whatever governs how much Google crawls a new site, repeatedly re-reading its sitemap is not part of it.

Being crawled is not being indexed

The gap between crawling and indexing is the number that matters, and it is stark. As of 2026-08-12, Search Console reported 48 URLs known for this site: 1 indexed, 17 crawled but not indexed, and 30 not yet processed. Googlebot has read the pages. It has decided not to keep them.

That distinction is worth internalising before you spend a week on technical SEO. Crawl access is the cheap part — robots.txt, a sitemap, and clean HTML get you that in a day. Whether anything is kept is a judgement about the content, and no amount of crawl plumbing changes it.

The traffic side reflects the same thing from the other end. Over the same period our top search referrers were duckduckgo.com (31), google.com (10), bing.com (9), kagi.com (7), search.brave.com (6). The engine that indexed the least is not the one sending the most.

Every crawler we saw

Every request to this site between 2026-07-19 and 2026-08-12 (25 days), read from our own nginx access logs. “Operated by” names the company the user agent claims; crawlers marked as training or answer engines collect for language models rather than a search index.

What we would tell a new site owner

Three things follow from this data, and none of them is a growth hack.

Read your own logs before you read advice about logs. The ratios here were not what we expected, and they were free to obtain. Whatever your site is, the answer for your site is already sitting in /var/log/nginx.

Decide deliberately about AI crawlers. They are the majority of crawl load in this sample. robots.txt can allow or disallow them individually — that is a real choice with real trade-offs, not a default to drift into.

Do not read crawl volume as progress. Being crawled thousands of times told us nothing about whether the content was worth keeping. The indexing numbers answered that, and they answered it differently.

FAQ

Do these numbers apply to my site?

Not directly. This is one small, new, English-language technical site with no backlink profile, measured over 25 days. Treat the ratios as a prompt to check your own logs, not as a benchmark.

Can user agents be faked?

Yes. User agents are self-reported. These counts describe traffic that identified itself as each crawler. Verifying identity requires reverse-DNS or published IP ranges, which we did not apply here — so treat named counts as upper bounds.

Does blocking AI crawlers help SEO?

There is no evidence in this data either way. Blocking a language-model crawler changes who can read your content; it does not change how a search engine judges it. Decide it on licensing and bandwidth grounds, not ranking ones.

Why is crawled-but-not-indexed so common on new sites?

Google states that crawling and indexing are separate steps and that it does not index every page it crawls. On a new site with little established value, the not-indexed bucket is where most pages sit until that changes.