Bots are now most of your traffic: what that means for your site

How much of the web is now bots?

Bots are now the majority of web traffic. Imperva’s 2025 Bad Bot Report says automated traffic reached 51% of all web traffic, the first time in a decade it passed human activity. The same report puts bad bots at 37% of all internet traffic.

Imperva also says attackers now use AI to build bots that go after APIs, exploit business logic and commit fraud, and that AI is lowering the skill needed to start. The report does not give a share for AI-built bots, so treat the direction as clear and the exact size as unknown.

For a small business, the practical point is simple. Some of your visitors, bandwidth and server load go to software, not customers. Your analytics may hide this, because many analytics tools filter bots out, while your server logs and hosting bill do not.

What kinds of bots visit a business website?

A bot is any program that requests pages without a person clicking. Cloudflare’s verified bot categories give a useful split, and the table below groups them by what they mean for you.

Bot type	What it does	Example	Usually
Search engine crawler	Indexes pages so people can find you	Googlebot, Bingbot	Welcome
AI search crawler	Indexes pages for AI answer features	OAI-SearchBot	Welcome if you want AI citations
AI training crawler	Collects content to train AI models	GPTBot, ClaudeBot	Your choice
AI assistant (user-triggered)	Opens a page because a person asked a question	ChatGPT-User, Perplexity-User	Mostly welcome
Monitoring and preview bots	Check uptime, build link previews, fetch feeds	Uptime monitors, Slack link previews	Welcome
Scrapers	Copy prices, text or images in bulk	Unnamed, often disguised	Usually unwelcome
Credential stuffers	Try leaked passwords on your login page	Unnamed, spread across many IPs	Unwelcome
Vulnerability scanners	Probe for known weak software	Unnamed	Unwelcome when not yours

The first four rows are named, published and (for the big vendors) verifiable. The last three are the “bad bots” described in the glossary entry. They do not announce themselves and often copy a browser’s identity.

How do AI crawlers differ from older bots?

AI crawlers differ mainly in purpose, and the vendors now publish separate names for each purpose. That lets you treat them differently.

• OpenAI’s crawler documentation lists GPTBot for model training, OAI-SearchBot for ChatGPT search results, and ChatGPT-User for pages opened on a user’s request.
• Anthropic’s support article describes ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for user requests.
• Perplexity’s documentation describes PerplexityBot as a search crawler that is not used for training, and Perplexity-User as a fetcher that runs when someone asks a question.

Blocking all of them in one move can remove you from AI answers as well as from training sets. The guide to blocking AI crawlers explains which to allow and gives a robots.txt you can copy.

How do bad bots show up in your logs and bills?

Each type of bad bot leaves a recognisable pattern, and you can spot most of them without special tools.

• Scrapers: a long run of page requests in a short time, often walking through every product or article, with a browser-like identity but no loaded images or scripts.
• Credential stuffing: many failed requests to your login or checkout page, spread across many IP addresses so no single address looks busy. OWASP defines credential stuffing as trying username and password pairs stolen in a breach of a different site.
• Scanners: requests for paths your site does not have, such as admin files and old plugin folders, producing a burst of 404 errors.
• AI crawlers: steady requests under a named identity such as GPTBot or ClaudeBot, which you can match against each vendor’s published details.

The cost side is just as concrete. Extra requests raise bandwidth and server use, which can slow the site for real customers or push a hosting plan into its next tier. Floods of failed logins can also get real customers locked out.

What should you do first?

Start by finding out what you have, then act on the biggest item. Take these steps in order.

1. Ask your host or CDN for a bot or traffic report, or filter your server logs by user agent for the last 30 days.
2. Sort the traffic into the table categories above. Named crawlers go in one pile, unnamed bulk traffic in another.
3. Decide your AI policy for training crawlers and search crawlers separately, then set it in robots.txt and your CDN using the crawler blocking guide.
4. Protect your login page with rate limits and two-factor sign-in, as set out in the guide to stopping credential stuffing.
5. Put a firewall or CDN bot filter in front of the site if you do not have one, and review its reports monthly.

A named user agent proves nothing on its own, because anyone can type “GPTBot” into a script. Vendors publish IP ranges so you can check a claim. Anthropic’s article, for example, points to a published list of its IP ranges, and Common Crawl documents a reverse-DNS check for its CCBot crawler.