Build a web crawler with Queues and Brow·续

Build a web crawler with Queues and Brow·续

await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1, });

return { numCloudflareLinks, screenshot: ((await page.screenshot({ fullPage: true })) as Buffer).buffer, }; };


This helper function opens a new page in Puppeteer and navigates to the provided URL. `numCloudflareLinks` uses Puppeteer's `$$eval` (equivalent to `document.querySelectorAll`) to find the number of links to a `cloudflare.com` page. Checking if the link's `href` is to a `cloudflare.com` page is wrapped in a `try...catch` to handle cases where `href`s may not be URLs.

Then, the function sets the browser viewport size and takes a screenshot of the full page. The screenshot is returned as a `Buffer` so it can be converted to an `ArrayBuffer` and written to KV.

To enable recursively crawling links, add a snippet after checking the number of Cloudflare links to send messages recursively from the queue consumer to the queue itself. Recursing too deep, as is possible with crawling, will cause a Durable Object `Subrequest depth limit exceeded.` error. If one occurs, it is caught, but the links are not retried.

```ts
// const numCloudflareLinks = await page.$$eval("a", (links) => { ...

await page.$$eval("a", async (links) => {
    const urls: MessageSendRequest&lt;Message>[] = links.map((link) => {
        return {
            body: {
                url: link.href,
            },
        };
    });
    try {
        await env.CRAWLER_QUEUE.sendBatch(urls);
    } catch {} // do nothing, likely hit subrequest limit
});

// await page.setViewport({ ...

Then, in the queue handler, call crawlPage on the URL.

// in the `queue` handler:
// ...
if (!isAllowed) {
    message.ack();
    continue;
}

try {
    const { numCloudflareLinks, screenshot } = await crawlPage(url);
    const timestamp = new Date().getTime();
    const resultKey = `${encodeURIComponent(url)}-${timestamp}`;
    await env.CRAWLER_LINKS_KV.put(resultKey, numCloudflareLinks.toString(), {
        metadata: { date: timestamp },
    });
    await env.CRAWLER_SCREENSHOTS_KV.put(resultKey, screenshot, {
        metadata: { date: timestamp },
    });
    message.ack();
} catch {
    message.retry();
}

// ...

This snippet saves the results from crawlPage into the appropriate KV namespaces. If an unexpected error occurred, the URL will be retried and resent to the queue again.

Saving the timestamp of the crawl in KV helps you avoid crawling too frequently.

Add a snippet before checking robots.txt to check KV for a crawl within the last hour. This lists all KV keys beginning with the same URL (crawls of the same page), and check if any crawls have been done within the last hour. If any crawls have been done within the last hour, the message is ack'ed and not retried.

type KeyMetadata = {
  date: number;
};

// in the `queue` handler:
// ...
for (const message of batch.messages) {
  const sameUrlCrawls = await env.CRAWLER_LINKS_KV.list({
    prefix: `${encodeURIComponent(url)}`,
  });

let shouldSkip = false;
  for (const key of sameUrlCrawls.keys) {
    if (timestamp - (key.metadata as KeyMetadata)?.date < 60 * 60 * 1000) {
      // if crawled in last hour, skip
      message.ack();
      shouldSkip = true;
      break;
    }
  }
  if (shouldSkip) {
    continue;
  }

let isAllowed = true;
  // ...

The final script is included below.

import puppeteer, { BrowserWorker } from "@cloudflare/puppeteer";
import robotsParser from "robots-parser";

type Message = {
    url: string;
};

export interface Env {
    CRAWLER_QUEUE: Queue&lt;Message>;
    CRAWLER_SCREENSHOTS_KV: KVNamespace;
    CRAWLER_LINKS_KV: KVNamespace;
    CRAWLER_BROWSER: BrowserWorker;
}

type Result = {
    numCloudflareLinks: number;
    screenshot: ArrayBuffer;
};

type KeyMetadata = {
    date: number;
};

export default {
    async fetch(req, env, ctx): Promise&lt;Response> {
        // util endpoint for testing purposes
        await env.CRAWLER_QUEUE.send({ url: await req.text() });
        return new Response("Success!");
    },
    async queue(batch, env, ctx): Promise&lt;void> {
        const crawlPage = async (url: string): Promise&lt;Result> => {
            const page = await (browser as puppeteer.Browser).newPage();

await page.goto(url, {
                waitUntil: "load",
            });

const numCloudflareLinks = await page.$$eval("a", (links) => {
                links = links.filter((link) => {
                    try {
                        return new URL(link.href).hostname.includes("cloudflare.com");
                    } catch {
                        return false;
                    }
                });
                return links.length;
            });

// to crawl recursively - uncomment this!
            /*await page.$$eval("a", async (links) => {
        const urls: MessageSendRequest&lt;Message>[] = links.map((link) => {
          return {
            body: {
              url: link.href,
            },
          };
        });
        try {
          await env.CRAWLER_QUEUE.sendBatch(urls);
        } catch {} // do nothing, might've hit subrequest limit
      });*/

await page.setViewport({
                width: 1920,
                height: 1080,
                deviceScaleFactor: 1,
            });

return {
                numCloudflareLinks,
                screenshot: ((await page.screenshot({ fullPage: true })) as Buffer)
                    .buffer,
            };
        };

let browser: puppeteer.Browser | null = null;
        try {
            browser = await puppeteer.launch(env.CRAWLER_BROWSER);
        } catch {
            batch.retryAll();
            return;
        }

for (const message of batch.messages) {
            const { url } = message.body;
            const timestamp = new Date().getTime();
            const resultKey = `${encodeURIComponent(url)}-${timestamp}`;

const sameUrlCrawls = await env.CRAWLER_LINKS_KV.list({
                prefix: `${encodeURIComponent(url)}`,
            });

let shouldSkip = false;
            for (const key of sameUrlCrawls.keys) {
                if (timestamp - (key.metadata as KeyMetadata)?.date < 60 * 60 * 1000) {
                    // if crawled in last hour, skip
                    message.ack();
                    shouldSkip = true;
                    break;
                }
            }
            if (shouldSkip) {
                continue;
            }

let isAllowed = true;
            try {
                const robotsTextPath = new URL(url).origin + "/robots.txt";
                const response = await fetch(robotsTextPath);

const robots = robotsParser(robotsTextPath, await response.text());
                isAllowed = robots.isAllowed(url) ?? true; // respect robots.txt!
            } catch {}

if (!isAllowed) {
                message.ack();
                continue;
            }

try {
                const { numCloudflareLinks, screenshot } = await crawlPage(url);
                await env.CRAWLER_LINKS_KV.put(
                    resultKey,
                    numCloudflareLinks.toString(),
                    { metadata: { date: timestamp } },
                );
                await env.CRAWLER_SCREENSHOTS_KV.put(resultKey, screenshot, {
                    metadata: { date: timestamp },
                });
                message.ack();
            } catch {
                message.retry();
            }
        }

await browser.close();
    },
} satisfies ExportedHandler&lt;Env, Message>;

8. Deploy your Worker

To deploy your Worker, run the following command:

npmyarnpnpm

npx wrangler deploy

yarn wrangler deploy

pnpm wrangler deploy

You have successfully created a Worker which can submit URLs to a queue for crawling and save results to Workers KV.

To test your Worker, you could use the following cURL request to take a screenshot of this documentation page.

Test with a cURL requestbash

curl &lt;YOUR_WORKER_URL> \
  -H "Content-Type: application/json" \
  -d 'https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-run/'

Refer to the GitHub repository for the complete tutorial ↗︎, including a front end deployed with Pages to submit URLs and view crawler results.

Was this helpful?

YesNo

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-run/#page","headline":"Build a web crawler with Queues and Browser Run","description":"Example of how to use Queues and Browser Run to power a web crawler.","url":"https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-run/","inLanguage":"en","image":"https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-run/og.png?v=fc7a3d9b98f261c7","dateModified":"2026-10-12","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"},"keywords":["TypeScript"]}

description: Review the headers Cloudflare automatically attaches to every Browser Run request, including User-Agent and bot detection identifiers. title: Automatic request headers image: https://developers.cloudflare.com/browser-run/reference/automatic-request-headers/og.png?v=42ce85fa8a1425c6

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/browser-run/llms.txt
Use this file to discover all available pages before exploring further.

Build a web crawler with Queues and Brow·续

Build a web crawler with Queues and Brow·续

8. Deploy your Worker

Related resources

On this page