cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-19 16:53:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge branch 'main' into main

Browse source
This commit is contained in:
Crazyroostereye 2025-04-30 22:13:51 +02:00 • committed by GitHub
commit 6baa7725b3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 160 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

47
code/test_files/haproxy-block-ai-bots.txt Normal file
View file

@ -0,0 +1,47 @@
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
FacebookBot
facebookexternalhit
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
omgili
omgilibot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot
crawler.with.dots
star***crawler
Is this a crawler?
a[mazing]{42}(robot)
2^32$
curl|sudo bash