/crawl - Crawl web content

Last updated Oct 10, 2026 The /crawl endpoint scrapes content from a starting URL and follows links across the site, up to a configurable depth or page limit. Responses can be returned as HTML, Markdown, or JSON.

The /crawl endpoint is available via the REST API. Create a custom API Token with the Browser Run - Edit permission.

Endpoint

https://api.cloudflare.com/client/v4/accounts/<account_id>/browser-run/crawl

Required fields

• url (string)

Refer to optional parameters for additional customization options.

Common use cases

• Building knowledge bases or training AI systems (such as RAG applications) with up-to-date web content
• Scraping and analyzing content across multiple pages for research, summarization, or monitoring

How it works

There are two steps to using the /crawl endpoint:

1. Initiate the crawl job — A POST request where you initiate the crawl and receive a response with a job id.
2. Request results of the crawl job — Monitor the crawl with an event subscription or GET request. When it finishes, send a GET request for the results.

Crawl jobs have a maximum run time of seven days. If a job does not finish within this time, it will be cancelled due to timeout. Job results are available for 14 days after the job completes, after which the job data is deleted.

Free plan limitations

Users on the Workers Free plan are subject to additional crawl-specific restrictions. Refer to crawl endpoint limits for details.

Initiate the crawl job

Send a POST request with a url to start a crawl job. The API responds immediately with a job id you will use to retrieve results. Refer to optional parameters for additional customization options.

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://developers.cloudflare.com/workers/"
  }'

Example response:

{
    "success": true,
    "result": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e"
}

Request results of the crawl job

To check the status or request the results of your crawl job, use the job id you received:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl/c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e' \
  -H 'Authorization: Bearer YOUR_API_TOKEN'

The response includes a status field indicating the current state of the crawl job. The possible job statuses are:

• running — The crawl job is currently in progress.
• cancelled_due_to_timeout — The crawl job exceeded the maximum run time of seven days.
• cancelled_due_to_limits — The crawl job was cancelled because it hit account limits.
• cancelled_by_user — The crawl job was manually cancelled by the user.
• errored — The crawl job encountered an error.
• completed — The crawl job finished successfully.

Subscribe to lifecycle events

Use Queues event subscriptions as a push-based alternative to status polling. The linked reference includes setup instructions and payload schemas.

Browser Run is an account-level event source. A subscription receives events for all crawl jobs in your account:

• crawl.started — A crawl job starts.
• crawl.updated — A crawled URL changes status.
• crawl.finished — A crawl job finishes.

These events provide lifecycle and status information, not crawled page content. After crawl.finished, send a GET request with the job ID to fetch the full results.

Polling for completion

Since crawl jobs run asynchronously, you can poll the endpoint periodically to check when the job finishes. Add ?limit=1 to the request URL so the response stays lightweight — you only need the job status, not the full set of crawled records.

async function waitForCrawl(accountId, jobId, apiToken) {
    const maxAttempts = 60;
    const delayMs = 5000;

for (let i = 0; i < maxAttempts; i++) {
        const response = await fetch(
            `https://api.cloudflare.com/client/v4/accounts/${accountId}/browser-run/crawl/${jobId}?limit=1`,
            {
                headers: {
                    Authorization: `Bearer ${apiToken}`,
                },
            },
        );

const data = await response.json();
        const status = data.result.status;

if (status !== "running") {
            return data.result;
        }

await new Promise((resolve) => setTimeout(resolve, delayMs));
    }

throw new Error("Crawl job did not complete within timeout");
}

Once the job reaches a terminal status, fetch the full results without the limit parameter. You can also use the following query parameters to filter and paginate results:

• cursor — Cursor for pagination. If the response exceeds 10 MB, a cursor value will be included. Pass it as a query parameter to retrieve the next page of results.
• limit — Maximum number of records to return.
• status — Filter by URL status: queued, completed, disallowed, skipped, errored, or cancelled.

Example with query parameters:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl/c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e?cursor=10&limit=10&status=completed' \
  -H 'Authorization: Bearer YOUR_API_TOKEN'

Example response:

{
    "result": {
        "id": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e",
        "status": "completed",
        "browserSecondsUsed": 134.7,
        "total": 50,
        "finished": 50,
        "records": [
            {
                "url": "https://developers.cloudflare.com/workers/",
                "status": "completed",
                "markdown": "# Cloudflare Workers\nBuild and deploy serverless applications...",
                "metadata": {
                    "status": 200,
                    "title": "Cloudflare Workers · Cloudflare Workers docs",
                    "url": "https://developers.cloudflare.com/workers/"
                }
            },
            {
                "url": "https://developers.cloudflare.com/workers/get-started/quickstarts/",
                "status": "completed",
                "markdown": "## Quickstarts\nGet up and running with a simple Hello World...",
                "metadata": {
                    "status": 200,
                    "title": "Quickstarts · Cloudflare Workers docs",
                    "url": "https://developers.cloudflare.com/workers/get-started/quickstarts/"
                }
            }
            // ... 48 more entries omitted for brevity
        ],
        "cursor": 10
    },
    "success": true
}

Errored and blocked pages

If a crawled page returns an HTTP error (such as 402, 403, or 500), the record for that URL will have "status": "errored".

This information is only available in the crawl results (step 2) — the initiation response only returns the job id. Because crawl jobs run asynchronously, the crawler does not fetch page content at initiation time.

To view only errored records, filter by status=errored:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl/{job_id}?status=errored' \
  -H 'Authorization: Bearer YOUR_API_TOKEN'

The record's status field contains the HTTP status code returned by the origin server, and html contains the response body. This is useful for understanding site owners' intent when they block crawlers — for example, sites using AI Crawl Control ↗︎ may return a custom status code and message.

Cancel a crawl job

To cancel a crawl job that is currently in progress, use the job id you received:

curl -X DELETE 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl/c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e' \
  -H 'Authorization: Bearer YOUR_API_TOKEN'

A successful cancellation will return a 200 OK status code. The job status will be updated to cancelled, and all URLs that have been queued to be crawled will be cancelled.

Optional parameters

The following optional parameters can be used in your crawl request, in addition to the required url parameter. These are parameters specific to the /crawl endpoint.

When render is true (the default), crawl jobs also support all standard Browser Run parameters such as rejectResourceTypes, rejectRequestPattern, cookies, and setExtraHTTPHeaders. When render is false, only the crawl-specific parameters listed in the table below are supported. For the full list, refer to the API reference.

Pattern behavior

excludePatterns has strictly higher priority. If a URL matches an exclude rule, it is skipped, regardless of whether it matches an include rule.

• No rules — Everything is indexed.
• Exclude only — Everything is indexed except items matching the exclude patterns.
• Include only — Only items matching the include patterns are indexed; everything else is ignored.

Viewing skipped URLs

The skipped status applies to URLs that the crawler discovered and evaluated individually, but then chose not to fetch because they were excluded by your crawl configuration, such as includeExternalLinks, includeSubdomains, or includePatterns/excludePatterns. To view these URLs, query the crawl job results with status=skipped.

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl/{job_id}?status=skipped' \
  -H 'Authorization: Bearer YOUR_API_TOKEN'

Because skipped only tracks URLs that were evaluated one by one, it applies mainly to crawls that use source: links (or source: all), where the crawler follows links scraped from pages and checks each one against your configuration. When crawling from a sitemap (source: sitemaps), URLs that fall outside your configuration are filtered out in bulk before they are evaluated individually, so they are omitted from the job entirely rather than recorded as skipped. As a result, the set of skipped URLs is not an exhaustive list of every URL that was left out of the crawl.

render parameter

If you use render: true, which is the default, the crawl endpoint spins up a headless browser and executes page JavaScript. If you use render: false, the crawl endpoint does a fast HTML fetch without executing JavaScript.

Use render: true when the page builds content in the browser. Use render: false when the content you need is already in the initial HTML response.

Crawls that use render: true use a headless browser and are billed under typical Browser Run pricing. Crawls that use render: false run on Workers instead of a headless browser. During the beta, render: false crawls are not billed. After the beta, they will be billed under Workers pricing.

Example with all optional parameters

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://www.exampledocs.com/docs/",
    "crawlPurposes": ["search"],
    "contentUse": "reference",
    "limit": 50,
    "depth": 2,
    "formats": ["markdown"],
    "render": false,
    "maxAge": 7200,
    "modifiedSince": 1704067200,
    "source": "all",
    "options": {
      "includeExternalLinks": true,
      "includeSubdomains": true,
      "includePatterns": [
        "**/api/v1/*"
      ],
      "excludePatterns": [
        "*/learning-paths/*"
      ]
    }
}'

Advanced usage

Looking for more parameters?

Visit the Browser Run API reference for all available parameters, such as setting HTTP credentials using authenticate, setting cookies, and customizing load behavior using gotoOptions.

Documentation site crawl

Crawl only documentation pages and exclude specific sections:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/docs",
    "limit": 200,
    "depth": 5,
    "formats": ["markdown"],
    "options": {
      "includePatterns": [
        "https://example.com/docs/**"
      ],
      "excludePatterns": [
        "https://example.com/docs/changelog/**",
        "https://example.com/docs/archive/**"
      ]
    }
  }'

Product catalog extraction with AI

Extract structured product data using the json format. This leverages Workers AI by default. Refer to the /json endpoint to learn more.

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-run/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://shop.example.com/products",
    "limit": 50,
    "formats": ["json"],
    "jsonOptions": {
      "prompt": "Extract product name, price, description, and availability",
      "response_format": {
        "type": "json_schema",
        "json_schema": {
          "name": "product",
          "properties": {
            "name": "string",
            "price": "number",
            "currency": "string",
            "description": "string",
            "inStock": "boolean"
          }
        }
      }
    },
    "options": {
      "includePatterns": [
        "https://shop.example.com/products/*"
      ]
    }
  }'