AI bots 'strip‑mining' web
Cloudflare-backed analysis cited by Business Insider shows major AI crawlers are scraping large volumes of public web content while sending little referral traffic back to publishers. The report singled out Anthropic among the biggest crawlers in the pattern it describes as 'strip-mining' the web. (businessinsider.com)
AI companies are sending bots across the web to copy public pages at a scale that far exceeds the traffic they send back to publishers. Cloudflare’s tracking shows Anthropic among the most lopsided examples. (blog.cloudflare.com) Cloudflare said on July 1, 2025 that it began publishing a crawl-to-refer ratio, which compares how often a platform’s bots request HTML pages with how often users later click through from that platform to the publisher. The same day, the company said Anthropic traffic made it “30,000 times more difficult” for creators to get visits than the old Google search model. (blog.cloudflare.com 1) (blog.cloudflare.com 2) Cloudflare’s later analysis said Anthropic’s ratio improved but still remained the highest among major AI platforms in July 2025, at about 38,000 crawls for each referral visit. The company said OpenAI was around 1,091 to 1 in July 2025, while Google search was far lower than the main AI crawlers. (blog.cloudflare.com) The dispute is about the bargain that built search: websites let engines index pages, and engines sent readers back. Cloudflare said artificial intelligence systems often answer with summaries or excerpts that reduce the need to click through to the original page. (blog.cloudflare.com 1) (blog.cloudflare.com 2) Cloudflare turned that complaint into a product fight on July 1, 2025, when it said new sites on its network would block AI crawlers by default unless owners granted permission. It also announced a “Pay Per Crawl” system so publishers could charge bot operators for access. (cloudflare.com) (blog.cloudflare.com) Anthropic says site owners can block its crawler in `robots.txt`, and its help page says publishers can disallow ClaudeBot entirely or slow it with a crawl-delay setting. Cloudflare’s bot directory lists three Anthropic agents: ClaudeBot for crawling, Claude-SearchBot for search, and Claude-User for assistant access. (support.claude.com) (developers.cloudflare.com) That opt-out system is part of the tension. Cloudflare and publishers are pushing for explicit permission and payment, while Anthropic and other AI firms still rely on public-web access rules that have historically been governed by `robots.txt` and bot identification. (cloudflare.com) (support.claude.com) Business Insider framed the pattern as web “strip-mining,” using Cloudflare-backed numbers to argue that publishers are providing raw material while getting little audience in return. The next fight is whether blocking and paywalls for bots become standard across the public web. (businessinsider.com)
