cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-07 04:17:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions1

Add Cloudflares first-party scraper blocking to FAQ

Browse source
This commit is contained in:
Joshua Sheard 2024-08-04 21:38:02 +01:00 committed by GitHub
parent c7b781034e
commit 8dbbdbf44c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions

1
FAQ.md
View file

@ -23,6 +23,7 @@ That depends on your stack.
- Netlify
- [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman
- Cloudflare
- [Block AI bots, scrapers and crawlers with a single click](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click) by Cloudflare
- [I’m blocking AI crawlers](https://roelant.net/en/2024/im-blocking-ai-crawlers-part-2/) by Roelant
## Why should we block these crawlers?