cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-17 16:03:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

HAProxy converter added.

Browse source
This commit is contained in:
Rik Wijnen 2025-04-28 08:42:52 +02:00
parent c6c7f1748f
commit 50e739dd73
3 changed files with 75 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

57
haproxy-block-ai-bots.txt Normal file
View file

@ -0,0 +1,57 @@
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot