Overview
- Cloudflare announced on July 1 that it will change default settings so Search crawlers remain allowed while Training and Agent crawlers are blocked by default on pages that show ads, with the new defaults taking effect on September 15, 2026 unless site owners change their settings.
- The company is evolving its Pay Per Crawl product into a Pay Per Use model that pays publishers when their content is actually used in AI answers and is piloting that approach with Ceramic.ai and You.com.
- Cloudflare introduced new tooling for publishers, including an Attribution Business Insights dashboard, Answer Engine Optimization features, and content signals to tell crawlers when a page has changed, which it says will reduce wasted re‑fetching of unchanged pages.
- A core technical requirement is that AI firms must separate crawler intent for Search, Agent, and Training so site owners can allow indexing without permitting model training or agent use, and Cloudflare will block mixed‑use bots that do not separate traffic.
- Observers warn the plan could concentrate control over identification, measurement, permissions, and payments at Cloudflare and may favor large AI platforms that can pay or integrate, while smaller developers and standards for fair attribution remain unresolved.