AI Crawl Control (2)

Last updated Aug 14, 2026 Available on all plans

Monitor and control how AI services access your website content.

AI companies use web content to train their models and power AI applications. AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content, and provides tools to manage access according to your preferences.

With AI Crawl Control, you can:

  • See which AI services access your content - Monitor the dashboard to see crawler activity and request patterns
  • Control access with granular policies - Set allow or block rules for individual crawlers
  • Monitor robots.txt compliance - Track which crawlers follow your directives and create enforcement rules
  • Explore monetization options - Set up pay per crawl pricing for content access (private beta)
  • Deploy with zero configuration - Works automatically on all Cloudflare plans

Get started

Features

Manage AI crawlers

Control how AI crawlers interact with your domain.

Manage AI crawlers

Analyze AI traffic

Gain insight into how AI crawlers are interacting with your pages.

Analyze AI traffic

Track robots.txt

Track the health of robots.txt files and identify which crawlers are violating your directives.

Track robots.txt

Pay Per Crawl

Allow AI crawlers to access content by paying per crawl.

Pay per crawl

Use cases

Publishers and content creators

Publishers and content creators can monitor which AI crawlers are accessing their articles and educational content. Set policies to allow beneficial crawlers while blocking others.

E-commerce and business sites

E-commerce and business sites can identify AI crawler activity on product pages and business information. Control access to sensitive data like pricing and inventory.

Documentation sites

Documentation sites can track how AI crawlers are accessing their technical documentation. Gain insight into how AI crawlers are engaging with your site.

Bots

Identify and mitigate automated traffic to protect your domain from bad bots.

Web Application Firewall

Get automatic protection from vulnerabilities and the flexibility to create custom rules.

Analytics

View and analyze traffic on your domain.

Was this helpful?

YesNo

On this page

AI Crawl Control (2) - 图1Docs

{"@context":"https://schema.org","@type":"WebPage","@id":"https://developers.cloudflare.com/ai-crawl-control/#page","headline":"AI Crawl Control","description":"Monitor and control how AI services access your website content.","url":"https://developers.cloudflare.com/ai-crawl-control/","inLanguage":"en","image":"https://developers.cloudflare.com/ai-crawl-control/og.png?v=9f8a355b477ca97e","dateModified":"2026-08-14","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"},"keywords":["AI"]}

description: Observe and control your AI applications with analytics, caching, rate limiting, and model fallback through AI Gateway. title: Cloudflare AI Gateway image: https://developers.cloudflare.com/ai-gateway/og.png?v=d9b4c0d9dbab4986

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/ai-gateway/llms.txt
Use this file to discover all available pages before exploring further.