Skip to content
Cloudflare Docs

Direct AI crawlers with managed robots.txt

Protect your website or application from AI crawlers by implementing a robots.txt file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.

Cloudflare's managed robots.txt explicitly disallows known bots engaged in scraping for AI purposes.

AI bots are expected to follow the robots.txt directives.

Compatibility with existing robots.txt files

Cloudflare will independently check whether your website has an existing robots.txt file and update the behavior of this feature based on your website.

Existing robots.txt file

If your website already has a robots.txt file — verified by a HTTP 200 response — Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response.

For example, without this feature enabled, the robots.txt content of crawlstop.com would be:

User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest


Sitemap: https://www.crawlstop.com/sitemap.xml

With the managed robots.txt enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://crawlstop.com/robots.txt ↗.

No robots.txt file

If your website does not have a robots.txt file, Cloudflare creates a new file with our managed block directives and serves it for you.

Implementation

To implement a robots.txt file on your domain:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
  2. Go to Security > Bots.
  3. Select Configure Bot Fight Mode.
  4. Turn Manage bot traffic with robots.txt on.

Availability

Managed robots.txt for AI crawlers is available on all plans.