Pay Per Crawl FAQ

Last updated Apr 23, 2026

Frequently asked questions for site owners

Can I set different prices for different AI crawlers?

No. Pay per crawl allows you to configure different actions (Block, Charge, or Allow) for each crawler, but you can only set a single price that applies to all crawlers configured with the "Charge" option.

Frequently asked questions for AI bot operators

Will I be charged for re-crawling the same page?

Yes. Every time your AI crawler accesses content on a website protected with pay per crawl, it will incur the cost set by the site owner. You should implement mechanisms within your crawler to track expenditure and enforce any spending limits you want to set.

Some paths are always free to crawl. These paths are: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, /crawlers.json.

Am I charged for error responses?

No. Charging events are only triggered for successful HTTP response codes. Error responses are not billed, even if you have sent the crawler-exact-price or crawler-max-price headers.

What user agent should I use?

Use the standard user agents associated with your AI crawler that you have onboarded to Cloudflare and identified through Web Bot Auth.

Was this helpful?

YesNo

On this page

Pay Per Crawl FAQ - 图1Docs

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/#page","headline":"Pay Per Crawl FAQ","description":"Answers to common Pay Per Crawl questions.","url":"https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/","inLanguage":"en","image":"https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/og.png?v=7fc162f3e8f0901d","dateModified":"2026-04-23","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}

description: Connect your Stripe account for Pay Per Crawl payments. title: Connect Stripe image: https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/connect-to-stripe/og.png?v=4f3eb642dc786cfe

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/ai-crawl-control/llms.txt
Use this file to discover all available pages before exploring further.