Select Language:
Cloudflare has introduced a new Content Signals Policy, adding a feature to robots.txt that allows website owners to specify their preferences regarding how their content can be used after it has been accessed. The primary goal is to prevent Google from incorporating your content in AI Overviews, although this relies on Google’s compliance, which they have not yet committed to.
In July, Cloudflare’s CEO mentioned their efforts to get Google to restrict content usage specifically for AI Overviews, and this new policy may be part of that initiative. Cloudflare claims this feature enables site owners to block their content from appearing in AI Overviews while still remaining visible in traditional Google Search results.
The company explains that with this update, website publishers can modify their robots.txt files—simple instructions that AI companies are expected to follow—to specify, for example, that Google can use their content for search purposes but not for AI Overviews. It’s important to note, however, that robots.txt directives are not legally binding, and Cloudflare’s executives have acknowledged this.
While Cloudflare has given Google a heads-up about this feature, Google has not formally indicated whether they will honor these instructions. If AI companies ignore robots.txt directives, publishers might have grounds for legal action.
For users on the free plan, the Content Signals Policy can be disabled within the Security Settings section of the Cloudflare dashboard or through the Overview section.
This approach is summarized by industry experts noting that the policy allows site owners to add directives to their robots.txt files, distinguishing content use for AI training from search purposes. However, adherence by Google is not guaranteed.
ChatGPT
Perplexity
Gemini AI
Grok AI
