AI Crawl Control with Cloudflare Bots

Last updated Jul 1, 2026 AI Crawl Control works alongside other Cloudflare products, such as Cloudflare bot solutions. Bot solutions identifies traffic matching patterns of known bots, and can challenge or block the bots as you wish.

Order of precedence

  • AI Crawl Control's AI crawler blocking uses WAF custom rules, which take place before Cloudflare bot solutions.
  • AI Crawl Control's pay per crawl takes place after Cloudflare bot solutions.
graph LR
A[Traffic] --> B[WAF custom rules<br>AI Crawl Control: Crawler blocks]
B --> C[Cloudflare<br>Bot Solutions]
C --> D[AI Crawl Control:<br>Pay Per Crawl]
classDef highlight fill:#F6821F,color:white

For more information on how Cloudflare classifies bot traffic, refer to AI bots.

Examples

Consider the following examples.

Bot rule which blocks all AI bots vs pay per crawl

You may have both of the following enabled:

  • A selection of AI crawlers to be charged through AI Crawl Control's pay per crawl
  • Bot configuration option to Block AI Bots.

Since pay per crawl happens after bot solutions, you need to first turn off Block AI Bots to ensure pay per crawl works as intended.

Was this helpful?

YesNo

On this page

AI Crawl Control with Cloudflare Bots - 图1Docs

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-bots/#page","headline":"AI Crawl Control with Cloudflare Bots","description":"Use AI Crawl Control alongside bot management.","url":"https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-bots/","inLanguage":"en","image":"https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-bots/og.png?v=e219776dd2659fe6","dateModified":"2026-07-01","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}

description: Add licensing headers to crawler responses using Transform Rules. title: AI Crawl Control with Transform Rules image: https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-transform-rules/og.png?v=b1744fa001797be7

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/ai-crawl-control/llms.txt
Use this file to discover all available pages before exploring further.