Cloudflare adds one-call crawl and faster isolates
Cloudflare rolled out a /crawl endpoint that fetches a site in HTML, Markdown or JSON with a single API call, removing the need to manage browsers or scripts for basic crawling. It also launched Dynamic Workers, a way to run AI-generated code in lightweight isolates that the company says are 100x faster than traditional containers. (x.com/CloudflareDev/status/2031488099725754821, x.com/Cloudflare/status/2036429081214886183)
Cloudflare has added two new building blocks for developers: a crawl API that maps a site with one request, and Dynamic Workers for running generated code in isolates. (developers.cloudflare.com, blog.cloudflare.com) The new `/crawl` endpoint is part of Cloudflare’s Browser Rendering service. Cloudflare says it starts from one URL, follows links across the same site, and returns results as HTML, Markdown, or structured JavaScript Object Notation, or JSON. (developers.cloudflare.com, developers.cloudflare.com) Cloudflare published the crawl endpoint in a March 10, 2026 changelog entry and says developers can control depth, page limits, sitemap use, and load behavior through the application programming interface. The company also says the endpoint identifies itself as a bot and does not bypass bot checks or captchas. (developers.cloudflare.com, developers.cloudflare.com, developers.cloudflare.com) Browser rendering is the practice of loading a page the way a browser would, so scripts run and page content appears before extraction. Cloudflare’s existing Browser Rendering tools already offered single-page actions like fetching HTML, extracting Markdown, taking screenshots, generating portable document format files, and scraping elements. (developers.cloudflare.com, developers.cloudflare.com) Dynamic Workers address a different problem: how to run code written on the fly by an artificial intelligence model without giving that code direct access to an app’s main environment. In a March 24, 2026 post, Cloudflare said the system runs that code inside lightweight isolates and starts about 100 times faster than traditional containers, with startup measured in milliseconds. (blog.cloudflare.com) Cloudflare framed Dynamic Workers as a sandbox for artificial intelligence agents, which are software systems that call tools and execute tasks on a user’s behalf. The company said many developers have been using containers for that isolation layer, but argued containers add more startup delay and operational overhead. (blog.cloudflare.com) The two launches fit Cloudflare’s broader push to make its edge platform do more of the work that developers once handled with separate browser farms, scripts, and infrastructure. Its Browser Rendering docs describe a globally distributed service for headless browsers, while the Dynamic Workers post ties isolates directly to artificial intelligence workloads. (developers.cloudflare.com, blog.cloudflare.com) For developers building crawlers, research agents, or retrieval pipelines, the pitch is simpler plumbing: one endpoint to gather site content, and one sandbox to run generated code closer to users. Cloudflare’s claim is that both pieces can now run on the same platform, with fewer moving parts than the older browser-plus-container stack. (developers.cloudflare.com, blog.cloudflare.com)
