Changelog (18)

Last updated Apr 23, 2026

2026-06-16

Pay Per Crawl advanced configuration

You can now configure advanced Pay Per Crawl settings for your zone, including:

  • Disable Pay Per Crawl by URI pattern using Configuration Rules to offer free access to specific pages while charging for others.
  • Dynamic pricing by having your origin return a crawler-price response header, or by using a Cloudflare Worker to set prices based on request properties.

When dynamic pricing is enabled, Pay Per Crawl adds a cf-pay-per-crawl request header to origin requests so your origin or Worker can determine the appropriate price.

Refer to the Advanced configuration documentation for details.

2026-04-17

Introducing Redirects for AI Training

Cloudflare's network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the original page normally.

This feature leverages your existing <link rel="canonical"> tags. No additional configuration required beyond enabling the toggle. Available on Pro, Business, and Enterprise plans at no additional cost.

Refer to the Redirects for AI Training documentation for details.

2026-04-17

Tools to prepare your site for the agentic Internet

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.

Content Format insights

The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.

Directives tab (formerly Robots.txt)

The Robots.txt tab has been renamed to Directives and now includes a link to check your site's Agent Readiness ↗︎ score.

Refer to our blog post on preparing for the agentic Internet ↗︎ for more on why these capabilities matter.

2026-03-24

Advanced WAF customization for AI Crawl Control blocks

AI Crawl Control now supports extending the underlying WAF rule with custom modifications. Any changes you make directly in the WAF custom rules editor — such as adding path-based exceptions, extra user agents, or additional expression clauses — are preserved when you update crawler actions in AI Crawl Control.

If the WAF rule expression has been modified in a way AI Crawl Control cannot parse, a warning banner appears on the Crawlers page with a link to view the rule directly in WAF.

For more information, refer to WAF rule management.

2026-02-09

Analytics enhancements

AI Crawl Control metrics have been enhanced with new views, improved filtering, and better data visualization.

AI Crawl Control path patterns

Path pattern grouping

  • In the Metrics tab > Most popular paths table, use the new Patterns tab that groups requests by URI pattern ( /blog/*, /api/v1/*, /docs/*) to identify which site areas crawlers target most. Refer to the screenshot above.

Enhanced referral analytics

  • Destination patterns show which site areas receive AI-driven referral traffic.
  • In the Metrics tab, a new Referrals over time chart shows trends by operator or source.

Data transfer metrics

  • In the Metrics tab > Allowed requests over time chart, toggle Bytes to show bandwidth consumption.
  • In the Crawlers tab, a new Bytes Transferred column shows bandwidth per crawler.

Image exports

  • Export charts and tables as images for reports and presentations.

Learn more about analyzing AI traffic.

2026-02-04

New reference documentation

New reference documentation is now available for AI Crawl Control:

  • GraphQL API reference — Query examples for crawler requests, top paths, referral traffic, and data transfer. Includes key filters for detection IDs, user agents, and referrer domains.
  • Bot reference — Detection IDs and user agents for major AI crawlers from OpenAI, Anthropic, Google, Meta, and others.
  • Worker templates — Deploy the x402 Payment-Gated Proxy to monetize crawler access or charge bots while letting humans through free.

2026-01-13

AI Crawl Control Read Only role now available

Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level.

Users with this role can view the Overview, Crawlers, Metrics, Robots.txt, and Settings tabs but cannot modify crawler actions or settings.

This role is specific for AI Crawl Control. You still require correct permissions to access other areas / features of the dashboard.

To assign, go to Manage Account > Members and add a policy with the AI Crawl Control Read Only role scoped to the desired domain.

2025-12-18

New AI Crawl Control Overview tab

The Overview tab is now the default view in AI Crawl Control. The previous default view with controls for individual AI crawlers is available in the Crawlers tab.

What's new

  • Executive summary — Monitor total requests, volume change, most common status code, most popular path, and high-volume activity
  • Operator grouping — Track crawlers by their operating companies (OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta)
  • Customizable filters — Filter your snapshot by date range, crawler, operator, hostname, or path

AI Crawl Control Overview tab showing executive summary, metrics, and crawler groups

Get started

  1. Log in to the Cloudflare dashboard and select your account and domain.
  2. Go to AI Crawl Control, where the Overview tab opens by default with your activity snapshot.
  3. Use filters to customize your view by date range, crawler, operator, hostname, or path.
  4. Navigate to the Crawlers tab to manage controls for individual crawlers.

Learn more about analyzing AI traffic and managing AI crawlers.

2025-12-10

Pay Per Crawl (Private beta) - Discovery API, custom pricing, and advanced configuration

Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control.

For AI crawler operators

Discovery API

A new authenticated API endpoint allows verified crawlers to programmatically discover domains participating in Pay Per Crawl. Crawlers can use this to build optimized crawl queues, cache domain lists, and identify new participating sites. This eliminates the need to discover payable content through trial requests.

The API endpoint is GET https://crawlers-api.ai-audit.cfdata.org/charged_zones and requires Web Bot Auth authentication. Refer to Discover payable content for authentication steps, request parameters, and response schema.

Payment header signature requirement

Payment headers (crawler-exact-price or crawler-max-price) must now be included in the Web Bot Auth signature-input header components. This security enhancement prevents payment header tampering, ensures authenticated payment intent, validates crawler identity with payment commitment, and protects against replay attacks with modified pricing. Crawlers must add their payment header to the list of signed components when constructing the signature-input header.

New crawler-error header

Pay Per Crawl error responses now include a new crawler-error header with 11 specific error codes for programmatic handling. Error response bodies remain unchanged for compatibility. These codes enable robust error handling, automated retry logic, and accurate spending tracking.

For site owners

Configure free pages

Site owners can now offer free access to specific pages like homepages, navigation, or discovery pages while charging for other content. Create a Configuration Rule in Rules > Configuration Rules, set your URI pattern using wildcard, exact, or prefix matching on the URI Full field, and enable the Disable Pay Per Crawl setting. When disabled for a URI pattern, crawler requests pass through without blocking or charging.

Some paths are always free to crawl. These paths are: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, /crawlers.json.

Get started

AI crawler operators: Discover payable content | Crawl pages

Site owners: Advanced configuration

2025-11-10

Crawler drilldowns with extended actions menu

AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files.

What's new

Status code distribution chart

The Metrics tab includes a status code distribution chart showing HTTP response codes (2xx, 3xx, 4xx, 5xx) over time. Filter by individual crawler, category, operator, or time range to analyze how specific crawlers interact with your site.