Build a web crawler with Queues and Browser Run

Example of how to use Queues and Browser Run to power a web crawler.

Last updated Oct 12, 2026 This tutorial explains how to build and deploy a web crawler with Queues, Browser Run, and Puppeteer.

Puppeteer is a high-level library used to automate interactions with Chrome/Chromium browsers. On each submitted page, the crawler will find the number of links to cloudflare.com and take a screenshot of the site, saving results to Workers KV.

You can use Puppeteer to request all images on a page, save the colors used on a site, and more.

Prerequisites

  1. Sign up for a Cloudflare account ↗︎.
  2. Install Node.js ↗︎.

<details>

<summary>

Node.js version manager

</summary>

Use <a href="https://viteplus.dev/guide/env">Vite+ (<code>vp env</code>) ↗︎</a> to install and manage Node.js versions.

For supported Node.js versions, refer to the <a href="https://developers.cloudflare.com/workers/wrangler/install-and-update/#install-wrangler">Wrangler system requirements</a>.

</details>

1. Create new Workers application

To get started, create a Worker application using the create-cloudflare CLI ↗︎. Open a terminal window and run the following command:

npmyarnpnpm

npm create cloudflare@latest -- queues-web-crawler

yarn create cloudflare queues-web-crawler

pnpm create cloudflare@latest queues-web-crawler

For setup, select the following options:

  • For What would you like to start with?, choose Hello World example.
  • For Which template would you like to use?, choose Worker only.
  • For Which language do you want to use?, choose TypeScript.
  • For Do you want to use git for version control?, choose Yes.
  • For Do you want to add an AGENTS.md file to help AI coding tools understand Cloudflare APIs?, choose Yes.
  • For Do you want to deploy your application?, choose No (we will be making some changes before deploying).

Then, move into your newly created directory:

cd queues-web-crawler

2. Create KV namespace

We need to create a KV store. This can be done through the Cloudflare dashboard or the Wrangler CLI. For this tutorial, we will use the Wrangler CLI.

npmyarnpnpm

npx wrangler kv namespace create crawler_links

yarn wrangler kv namespace create crawler_links

pnpm wrangler kv namespace create crawler_links

npmyarnpnpm

npx wrangler kv namespace create crawler_screenshots

yarn wrangler kv namespace create crawler_screenshots

pnpm wrangler kv namespace create crawler_screenshots

🌀 Creating namespace with title "web-crawler-crawler-links"
✨ Success!
Add the following to your configuration file in your kv_namespaces array:
[[kv_namespaces]]
binding = "crawler_links"
id = "<GENERATED_NAMESPACE_ID>"

🌀 Creating namespace with title "web-crawler-crawler-screenshots"
✨ Success!
Add the following to your configuration file in your kv_namespaces array:
[[kv_namespaces]]
binding = "crawler_screenshots"
id = "<GENERATED_NAMESPACE_ID>"

Add KV bindings to the Wrangler configuration file

Then, in your Wrangler file, add the following with the values generated in the terminal:

{
    "kv_namespaces": [
        {
            "binding": "CRAWLER_SCREENSHOTS_KV",
            "id": "<GENERATED_NAMESPACE_ID>",
        },
        {
            "binding": "CRAWLER_LINKS_KV",
            "id": "<GENERATED_NAMESPACE_ID>",
        },
    ],
}

[[kv_namespaces]]
binding = "CRAWLER_SCREENSHOTS_KV"
id = "<GENERATED_NAMESPACE_ID>"

[[kv_namespaces]]
binding = "CRAWLER_LINKS_KV"
id = "<GENERATED_NAMESPACE_ID>"

3. Set up Browser Run

Now, you need to set up your Worker for Browser Run.

In your current directory, install Cloudflare's fork of Puppeteer and also robots-parser ↗︎:

npmyarnpnpmbun

npm i -D @cloudflare/puppeteer

yarn add -D @cloudflare/puppeteer

pnpm add -D @cloudflare/puppeteer

bun add -d @cloudflare/puppeteer

npmyarnpnpmbun

npm i robots-parser

yarn add robots-parser

pnpm add robots-parser

bun add robots-parser

Then, add a Browser Run binding. Adding a Browser Run binding gives the Worker access to a headless Chromium instance you will control with Puppeteer.

{
    "browser": {
        "binding": "CRAWLER_BROWSER",
    },
}

[browser]
binding = "CRAWLER_BROWSER"

4. Set up a Queue

Now, we need to set up the Queue.

npmyarnpnpm

npx wrangler queues create queues-web-crawler

yarn wrangler queues create queues-web-crawler

pnpm wrangler queues create queues-web-crawler

Outputtxt

Creating queue queues-web-crawler.
Created queue queues-web-crawler.

Add Queue bindings to Wrangler configuration

Then, in your Wrangler file, add the following:

{
    "queues": {
        "consumers": [
            {
                "queue": "queues-web-crawler",
                "max_batch_timeout": 60,
            },
        ],
        "producers": [
            {
                "queue": "queues-web-crawler",
                "binding": "CRAWLER_QUEUE",
            },
        ],
    },
}

[[queues.consumers]]
queue = "queues-web-crawler"
max_batch_timeout = 60

[[queues.producers]]
queue = "queues-web-crawler"
binding = "CRAWLER_QUEUE"

Adding the max_batch_timeout of 60 seconds to the consumer queue is important because it allows the Queue to collect messages into a batch over a longer period. This helps manage Browser Run rate limits and can improve efficiency by processing multiple URLs in a single batch with one browser instance.

Your final Wrangler file should look similar to the one below.

{
    "$schema": "./node_modules/wrangler/config-schema.json",
    "name": "web-crawler",
    "main": "src/index.ts",
    "upload_source_maps": true,
    // Set this to today's date
    "compatibility_date": "2026-10-27",
    "compatibility_flags": ["nodejs_compat"],
    "kv_namespaces": [
        {
            "binding": "CRAWLER_SCREENSHOTS_KV",
            "id": "<GENERATED_NAMESPACE_ID>",
        },
        {
            "binding": "CRAWLER_LINKS_KV",
            "id": "<GENERATED_NAMESPACE_ID>",
        },
    ],
    "browser": {
        "binding": "CRAWLER_BROWSER",
    },
    "queues": {
        "consumers": [
            {
                "queue": "queues-web-crawler",
                "max_batch_timeout": 60,
            },
        ],
        "producers": [
            {
                "queue": "queues-web-crawler",
                "binding": "CRAWLER_QUEUE",
            },
        ],
    },
}

"$schema" = "./node_modules/wrangler/config-schema.json"
name = "web-crawler"
main = "src/index.ts"
upload_source_maps = true
# Set this to today's date
compatibility_date = "2026-10-27"
compatibility_flags = [ "nodejs_compat" ]

[[kv_namespaces]]
binding = "CRAWLER_SCREENSHOTS_KV"
id = "<GENERATED_NAMESPACE_ID>"

[[kv_namespaces]]
binding = "CRAWLER_LINKS_KV"
id = "<GENERATED_NAMESPACE_ID>"

[browser]
binding = "CRAWLER_BROWSER"

[[queues.consumers]]
queue = "queues-web-crawler"
max_batch_timeout = 60

[[queues.producers]]
queue = "queues-web-crawler"
binding = "CRAWLER_QUEUE"

5. Add bindings to environment

Add the bindings to the environment interface in src/index.ts, so TypeScript correctly types the bindings. The queue is typed as Queue<Message>, where Message is defined in the following step.

import type { BrowserWorker } from "@cloudflare/puppeteer";

export interface Env {
    CRAWLER_QUEUE: Queue<Message>;
    CRAWLER_SCREENSHOTS_KV: KVNamespace;
    CRAWLER_LINKS_KV: KVNamespace;
    CRAWLER_BROWSER: BrowserWorker;
}

Add a fetch() handler to the Worker to submit links to crawl.

type Message = {
    url: string;
};

export interface Env {
    CRAWLER_QUEUE: Queue<Message>;
    // ... etc.
}

export default {
    async fetch(req, env, ctx): Promise<Response> {
        await env.CRAWLER_QUEUE.send({ url: await req.text() });
        return new Response("Success!");
    },
} satisfies ExportedHandler<Env>;

This will accept requests to any subpath and forwards the request's body to be crawled. It expects that the request body only contains a URL. In production, you should check that the request was a POST request and contains a well-formed URL in its body. This has been omitted for simplicity.

7. Crawl with Puppeteer

Add a queue() handler to the Worker to process the links you send.

import puppeteer from "@cloudflare/puppeteer";
import robotsParser from "robots-parser";

async queue(batch, env, ctx): Promise<void> {
  let browser: puppeteer.Browser | null = null;
  try {
    browser = await puppeteer.launch(env.CRAWLER_BROWSER);
  } catch {
    batch.retryAll();
    return;
  }

for (const message of batch.messages) {
    const { url } = message.body;

let isAllowed = true;
    try {
      const robotsTextPath = new URL(url).origin + "/robots.txt";
      const response = await fetch(robotsTextPath);

const robots = robotsParser(robotsTextPath, await response.text());
      isAllowed = robots.isAllowed(url) ?? true; // respect robots.txt!
    } catch {}

if (!isAllowed) {
      message.ack();
      continue;
    }

// TODO: crawl!
    message.ack();
  }

await browser.close();
},

This is a skeleton for the crawler. It launches the Puppeteer browser and iterates through the Queue's received messages. It fetches the site's robots.txt and uses robots-parser to check that this site allows crawling. If crawling is not allowed, the message is ack'ed, removing it from the Queue. If crawling is allowed, you can continue to crawl the site.

The puppeteer.launch() is wrapped in a try...catch to allow the whole batch to be retried if the browser launch fails. The browser launch may fail due to going over the limit for number of browsers per account.

```ts type Result = { numCloudflareLinks: number; screenshot: ArrayBuffer; };

const crawlPage = async (url: string): Promise<Result> => { const page = await (browser as puppeteer.Browser).newPage();

await page.goto(url, { waitUntil: "load", });

const numCloudflareLinks = await page.$$eval("a", (links) => { links = links.filter((link) => { try { return new URL(link.href).hostname.includes("cloudflare.com"); } catch { return false; } }); return links.length; });