/crawl - Crawl web content·续

Fast static content fetch

Fetch static HTML without rendering for faster crawling of static sites:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "render": false,
    "formats": ["html", "markdown"]
  }'

Crawl with authentication

Crawl pages behind HTTP authentication or with custom headers:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://secure.example.com",
    "limit": 50,
    "authenticate": {
      "username": "user",
      "password": "pass"
    }
  }'

You can also use cookies or custom headers for token-based authentication:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://api.example.com/docs",
    "limit": 100,
    "setExtraHTTPHeaders": {
      "X-API-Key": "your-api-key"
    }
  }'

Wait for dynamic content

Crawl single-page applications that load content dynamically:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://app.example.com",
    "limit": 50,
    "gotoOptions": {
      "waitUntil": "networkidle2",
      "timeout": 60000
    },
    "waitForSelector": {
      "selector": "[data-content-loaded]",
      "timeout": 30000,
      "visible": true
    }
  }'

Block unnecessary resources

Speed up crawling by blocking images and media. rejectResourceTypes is only available when render is true (the default).

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "rejectResourceTypes": [
      "image",
      "media",
      "font",
      "stylesheet"
    ]
  }'

Crawler behavior

How the crawler discovers URLs

The crawler discovers and processes URLs in the following order (when using source: all, the default):

  1. Starting URL — The URL specified in your request.
  2. Sitemap links — URLs found in the site's sitemap.
  3. Page links — Links scraped from pages, if not already found in the sitemap.

Use the source parameter to customize which sources the crawler uses. The available options are:

  • all — Uses both sitemaps and page links (default).
  • sitemaps — Only crawls URLs found in the site's sitemap.
  • links — Only crawls links found on pages, ignoring sitemaps.

robots.txt and bot protection

The /crawl endpoint respects the directives of robots.txt files, including crawl-delay. If a site does not specify a crawl-delay in its robots.txt, the crawler uses a default delay of 0.5 seconds between requests to the same domain to avoid overwhelming the origin server. All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed". For guidance on configuring robots.txt and sitemaps for sites you plan to crawl, refer to robots.txt and sitemaps. If you want to block the /crawl endpoint from accessing your site, refer to Blocking crawlers with robots.txt.

Bot protection may block crawling

Browser Run does not bypass CAPTCHAs, Turnstile challenges, or any other bot protection mechanisms. If a target site uses Cloudflare products that control or restrict bot traffic such as Bot Management, Web Application Firewall (WAF), or Turnstile, the same rules will apply to the Browser Run crawler.

If you are crawling your own site and want Browser Run to access it freely, you can create a WAF skip rule to allowlist Browser Run. Refer to Can I allowlist Browser Run on my own website? for instructions. The /crawl endpoint uses bot detection ID 128292352.

User-Agent

The /crawl endpoint uses CloudflareBrowserRenderingCrawler/1.0 as its User-Agent, which is different from other Quick Actions endpoints. This User-Agent is not customizable. Unlike other Quick Actions endpoints, the userAgent parameter is not supported on the /crawl endpoint.

For a full list of default User-Agent strings, refer to Automatic request headers.

Content Signals

The /crawl endpoint respects Content Signals ↗︎ directives found in a target site's robots.txt file. Content Signals are a way for site owners to express preferences about how their content can be used by automated systems. For more background, refer to Giving users choice with Cloudflare's new Content Signals Policy ↗︎.

A site owner can include a Content-Signal directive in their robots.txt to allow or disallow specific categories of use:

  • search — Building a search index and providing search results with links and excerpts.
  • ai-input — Inputting content into AI models at query time (for example, retrieval-augmented generation or grounding).
  • ai-train — Training or fine-tuning AI models.

For example, a robots.txt that allows search indexing but disallows AI training:

robots.txttxt

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

A site owner can also declare a use directive to express the maximum level at which their content may be used. The levels, from least to most permissive, are:

  • immediate — Ephemeral, single-response use, where content is not retained.
  • reference — Content may be retained, indexed, or cited.
  • full — Unrestricted use, including AI training.

For example, a robots.txt that limits use to reference:

robots.txttxt

User-Agent: *
Content-Signal: use=reference
Allow: /

How /crawl enforces Content Signals

The /crawl endpoint enforces both the yes/no purpose directives (search, ai-input, ai-train) and the use level directive.

Purpose directives

By default, /crawl declares all three purposes: ["search", "ai-input", "ai-train"]. If a target site sets any of those content signals to no, the crawl request will be rejected at initiation with a 400 Bad Request error unless you explicitly narrow your declared purposes using the crawlPurposes parameter to exclude the disallowed use.

This means:

  1. Site has no Content Signals — The crawl proceeds normally.
  2. Site has Content Signals, and all your declared purposes are allowed — The crawl proceeds normally.
  3. Site sets a content signal to no, and that purpose is in your crawlPurposes — The crawl request is rejected with a 400 error and the message Crawl disallowed by Content-Signal directive (purpose or use level).

To crawl a site that disallows AI training but allows search, set crawlPurposes to only the purposes you need:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "crawlPurposes": ["search"],
    "formats": ["markdown"]
  }'

In this example, because the operator declared only search as their purpose, the crawl will succeed even if the site sets ai-train=no.

Use level directive

The contentUse parameter declares the level at which you intend to use the crawled content. Allowed values, from least to most permissive, are reference and full. The default is full.

The immediate level is not accepted as a contentUse value because the /crawl endpoint stores crawled content, which is not compatible with ephemeral, single-response use.

A crawl is rejected when your declared contentUse level is more permissive than the site's declared use level. For example:

  1. Site sets use=full (or does not set use) — Any contentUse value is allowed.
  2. Site sets use=reference — A crawl with contentUse: "reference" is allowed, but the default contentUse: "full" is rejected.
  3. Site sets use=immediate — All crawls are rejected, because both reference and full exceed the declared level.

To crawl a site that sets use=reference, set contentUse to reference:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "contentUse": "reference",
    "formats": ["markdown"]
  }'

Note

Content Signals are trust-based. By setting crawlPurposes and contentUse, you are declaring to the site owner how you intend to use the crawled content.

Troubleshooting

Crawl job returns no results or all URLs are skipped

If your crawl job completes but returns an empty records array, or all URLs show skipped or disallowed status:

  • robots.txt blocking — The crawler respects robots.txt rules. The /crawl endpoint identifies itself as CloudflareBrowserRenderingCrawler/1.0. Check the target site's robots.txt file to verify this user agent is allowed. Blocked URLs appear with "status": "disallowed".
  • Pattern filters too restrictive — Your includePatterns may not match any URLs on the site. Try crawling without patterns first to confirm URLs are discoverable, then add patterns.
  • No links found — The starting URL may not contain links. Try using source: "sitemaps", increasing the depth parameter, or setting includeSubdomains or includeExternalLinks to true.

Crawl rejected by Content Signals

If your crawl request returns a 400 Bad Request with the message Crawl disallowed by Content-Signal directive (purpose or use level), the target site's robots.txt includes a Content-Signal directive that disallows one or more of your declared crawlPurposes, or a use directive that is more restrictive than your declared contentUse level. To resolve this, check the site's robots.txt for Content-Signal: entries:

  • If the site sets a purpose to no, set crawlPurposes to only the purposes you need. For example, if the site sets ai-train=no and you only need search indexing, use "crawlPurposes": ["search"].
  • If the site sets a use level, set contentUse to a level at or below it. For example, if the site sets use=reference, use "contentUse": "reference".

Refer to Content Signals for details.

Crawl job takes too long

If a crawl job remains in running status for an extended period:

  • Slow page loads — Pages with heavy JavaScript take longer to render. Use render: false if the content you need is in the initial HTML.
  • Rate limiting — The crawler enforces a per-domain rate limit to avoid overwhelming origin servers. If a site specifies a crawl-delay in its robots.txt, the crawler respects it. Otherwise, the crawler uses a default delay of 0.5 seconds between requests to the same domain. If you run multiple crawl jobs targeting the same domain, they share the same per-domain rate limit, which can cause all jobs to take longer than if each ran individually.
  • Unnecessary resources — Block resources that are not needed for content extraction using rejectResourceTypes (for example, image, media, font).

Crawl job cancelled due to limits

A cancelled_due_to_limits status means your account hit its browser time limit. Workers Free plan accounts are capped at 10 minutes of browser use per day. To resolve this:

  • Upgrade to a Workers Paid plan for higher limits.
  • Use render: false for static content to avoid consuming browser time.
  • Increase maxAge to use cached results where possible.
  • Reduce the limit parameter.

JSON extraction errors

If the json format returns null or empty results:

  • Provide a clear prompt — Be specific about what data to extract and where it appears on the page (for example, "Extract the product name, price, and description from the main product section").
  • Define a response schema — Use response_format with a JSON schema to enforce the expected output structure.
  • Use a custom model — If the default Workers AI model does not produce the desired results, use the custom_ai parameter to specify a different model. Refer to Using a custom model (BYO API Key) for details.

If you have questions or encounter other errors, refer to the Browser Run FAQ and troubleshooting guide.

Troubleshooting

If you have questions or encounter an error, see the Browser Run FAQ and troubleshooting guide.

Was this helpful?

YesNo

On this page

/crawl - Crawl web content·续 - 图1Docs

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#page","headline":"/crawl - Crawl web content","description":"Scrape and follow links across a website using the Browser Run /crawl endpoint, with configurable depth and output formats.","url":"https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/","inLanguage":"en","image":"https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/og.png?v=0071d0f11e705093","dateModified":"2026-10-10","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}

description: Extract structured JSON data from webpages using AI with the Browser Run /json endpoint. title: /json - Capture structured data using AI image: https://developers.cloudflare.com/browser-run/quick-actions/json-endpoint/og.png?v=b63f4f4b3045a308

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/browser-run/llms.txt
Use this file to discover all available pages before exploring further.