cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-03 19:03:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

add more concise message about verified bots

Browse source 
Co-authored-by: Glyn Normington <work@underlap.org>
This commit is contained in:
always-be-testing 2025-02-15 11:22:10 -05:00 committed by GitHub
parent af87b85d7f
commit 5b13c2e504
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
README.md
View file

@ -39,21 +39,7 @@ Alternatively, you can also subscribe to new releases with your GitHub account b
## Report abusive crawlers
If you use [Cloudflare's hard block](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click) alongside this list, you can report abusive crawlers that don't respect `robots.txt` [here](https://docs.google.com/forms/d/e/1FAIpQLScbUZ2vlNSdcsb8LyTeSF7uLzQI96s0BKGoJ6wQ6ocUFNOKEg/viewform).
## Cloudflare Verified Bots
If you are unable to make use of [Cloudflare's hard block](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click) and/or have WAF rules that use the `cf.bot_management.verified_bot` condition based on [Cloudflare's Verified Bots](https://radar.cloudflare.com/traffic/verified-bots), please note that the following AI web crawlers are considered verified bots by Cloudflare:
- Amazonbot
- Applebot
- CCBot
- ChatGPT-User
- DuckAssistBot
- GoogleOther
- GPTBot
- OAI-SearchBot
- PerplexityBot
- PetalBot
But even if you don't use Cloudflare's hard block, their list of [verified bots](https://radar.cloudflare.com/traffic/verified-bots) may come in handy.
## Additional resources
- [Blocking Bots with Nginx](https://rknight.me/blog/blocking-bots-with-nginx/) by Robb Knight