Cloudflare will score verified bots on whether they respect site rules

Cloudflare has started rating the ongoing behaviour of bots on its Verified bots list, not just checking who runs them. The company explained the system in a blog post on October 16, 2026.

To be verified today, a bot has to identify itself properly, do only what it says it does, and avoid abuse. Cloudflare says identity alone does not prove good behaviour, since a known crawler can still ignore what a site wants because of a bug, a bad configuration or on purpose. The new system builds a trust band for each bot over time from two first signals. One is whether the bot keeps to the rules published in a site's robots.txt file, looking at repeated patterns rather than one-off mistakes. The other is whether it slows down or pauses after a site answers with HTTP 429, the Too Many Requests response.

Cloudflare is using the scores internally for now to build up history. It plans to let all customers pick a trust threshold that decides how verified bot traffic is handled, and to publish the scores on Cloudflare Radar later. The company points out it removed Perplexity's crawler from the verified list in 2025 after it tried to get around blocks on sites that had opted out.

Why it matters

As AI crawlers and agents multiply, site owners get a practical way to reward polite bots and push back on ones that ignore their wishes.

Source: Cloudflare Blog