robots.txt and sitemaps

Last updated Jun 16, 2026 This page provides general guidance on configuring robots.txt and sitemaps for websites you plan to access with Browser Run.

robots.txt is advisory, not enforceable

The robots.txt standard is a voluntary protocol. Only well-behaved bots choose to follow it. Any client can send any User-Agent string, so robots.txt rules that target a specific User-Agent cannot prevent a spoofed or malicious client from accessing your site. To enforce access restrictions, use server-side controls such as a web application firewall (WAF), request header validation, or authentication.

To verify that a request originated from Cloudflare Browser Run specifically, use the cryptographic Web Bot Auth signatures or the non-configurable headers that Cloudflare attaches to every Browser Run request. You can also create WAF rules using the bot detection IDs to allow or block Browser Run traffic.

Identifying Browser Run requests

Requests can be identified by the automatic headers that Cloudflare attaches:

  • User-Agent: Each Browser Run method has a different default User-Agent, which you can use to write targeted robots.txt rules
  • cf-brapi-request-id: Unique identifier for Quick Actions requests
  • Signature-agent: Pointer to Cloudflare's bot verification keys

To allow or block Browser Run traffic using WAF rules instead of robots.txt, use the bot detection IDs on the automatic request headers page.

Best practices for robots.txt

A well-configured robots.txt helps crawlers understand which parts of your site they can access.

Reference your sitemap

Include a reference to your sitemap in robots.txt so crawlers can discover your URLs:

robots.txttxt

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps:

robots.txttxt

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/blog-sitemap.xml

Set a crawl delay

Use crawl-delay to control how frequently crawlers request pages from your server:

robots.txttxt

User-agent: *
Crawl-delay: 2
Allow: /

Sitemap: https://example.com/sitemap.xml

The value is in seconds. A crawl-delay of 2 means the crawler waits two seconds between requests.

Blocking crawlers with robots.txt

If you want to prevent Browser Run (or other crawlers) from accessing your site, you can configure your robots.txt to restrict access.

Block all bots from your entire site

To prevent all crawlers from accessing any page on your site:

robots.txttxt

User-agent: *
Disallow: /

This is the most restrictive configuration and blocks all compliant bots, not just Browser Run.

Block only the /crawl endpoint

The /crawl endpoint identifies itself with the User-Agent CloudflareBrowserRenderingCrawler/1.0. To block the /crawl endpoint while allowing all other traffic (including other Browser Run Quick Actions endpoints, which use a different User-Agent):

robots.txttxt

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: *
Allow: /

Block the /crawl endpoint on specific paths

To allow the /crawl endpoint to access your site but block specific sections:

robots.txttxt

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: *
Allow: /

Best practices for sitemaps

Structure your sitemap to help crawlers process your site efficiently:

sitemap.xmlxml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/important-page</loc>
    <lastmod>2025-01-15T00:00:00+00:00</lastmod>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/other-page</loc>
    <lastmod>2025-01-10T00:00:00+00:00</lastmod>
    <priority>0.5</priority>
  </url>
</urlset>
Attribute Purpose Recommendation
<loc> URL of the page Required. Use full URLs.
<lastmod> Last modification date Include to help the crawler identify updated content. Use ISO 8601 format.
<priority> Relative importance (0.0-1.0) Set higher values for important pages. The crawler will process pages in priority order.

Sitemap index files

For large sites with multiple sitemaps, use a sitemap index file. Browser Run uses the depth parameter to control how many levels of nested sitemaps are crawled:

sitemap.xmlxml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  ...
</urlset>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>https://www.example.com/sitemap-products.xml</loc>
   </sitemap>
   <sitemap>
      <loc>https://www.example.com/sitemap-blog.xml</loc>
   </sitemap>
</sitemapindex>

Caching headers

Browser Run periodically refetches sitemaps to keep content fresh. Serve your sitemap with Last-Modified or ETag response headers so the crawler can detect whether the sitemap has changed since the last fetch.

Recommendations

  • Include <lastmod> on all URLs to help identify which pages have changed. Use ISO 8601 format (for example, 2025-01-15T00:00:00+00:00).
  • Use sitemap index files for large sites with multiple sitemaps.
  • Compress large sitemaps using .gz format to reduce bandwidth.
  • Keep sitemaps under 50 MB and 50,000 URLs per file (standard sitemap limits).

Was this helpful?

YesNo

On this page

robots.txt and sitemaps - 图1Docs

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/browser-run/reference/robots-txt/#page","headline":"robots.txt and sitemaps","description":"Configure robots.txt rules and sitemaps to control how Browser Run accesses your website.","url":"https://developers.cloudflare.com/browser-run/reference/robots-txt/","inLanguage":"en","image":"https://developers.cloudflare.com/browser-run/reference/robots-txt/og.png?v=66e20b8e3106147c","dateModified":"2026-06-16","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}

description: View the list of pre-installed fonts available in the Browser Run Chromium environment for screenshots and PDFs. title: Supported fonts image: https://developers.cloudflare.com/browser-run/reference/supported-fonts/og.png?v=70fdd90cf35674f7

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/browser-run/llms.txt
Use this file to discover all available pages before exploring further.