cdransf/coryd.dev-eleventy
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: post revision

Browse source
This commit is contained in:
Cory Dransfeldt 2024-03-27 09:01:59 -07:00
parent 0c5c9fbd96
commit f5b3ebbacb
No known key found for this signature in database
1 changed files with 2 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

46
src/posts/2024/go-ahead-and-block-ai-web-crawlers.md
View file

@ -39,71 +39,27 @@ User-agent: *
Disallow: Disallow:
User-agent: AdsBot-Google User-agent: AdsBot-Google
Disallow: /
User-agent: Amazonbot User-agent: Amazonbot
Disallow: /
User-agent: anthropic-ai User-agent: anthropic-ai
Disallow: /
User-agent: AwarioRssBot User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot User-agent: AwarioSmartBot
Disallow: /
User-agent: Bytespider User-agent: Bytespider
Disallow: /
User-agent: CCBot User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai User-agent: cohere-ai
Disallow: /
User-agent: DataForSeoBot User-agent: DataForSeoBot
Disallow: /
User-agent: FacebookBot User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended User-agent: Google-Extended
Disallow: /
User-agent: GPTBot User-agent: GPTBot
Disallow: /
User-agent: ImagesiftBot User-agent: ImagesiftBot
Disallow: /
User-agent: magpie-crawler User-agent: magpie-crawler
Disallow: /
User-agent: omgili User-agent: omgili
Disallow: /
User-agent: omgilibot User-agent: omgilibot
Disallow: /
User-agent: peer39_crawler User-agent: peer39_crawler
Disallow: /
User-agent: peer39_crawler/1.0 User-agent: peer39_crawler/1.0
Disallow: /
User-agent: PerplexityBot User-agent: PerplexityBot
Disallow: /
User-agent: YouBot User-agent: YouBot
Disallow: / Disallow: /
``` ```
@ -112,4 +68,6 @@ Disallow: /
- [I’m blocking AI-crawlers](https://roelant.net/en/2023/im-blocking-ai-crawlers/) - [I’m blocking AI-crawlers](https://roelant.net/en/2023/im-blocking-ai-crawlers/)
- [Block the Bots that Feed “AI” Models by Scraping Your Website](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/) - [Block the Bots that Feed “AI” Models by Scraping Your Website](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/)
**Update March 27, 2024:** Many thanks to Jens for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement.
[^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations. [^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations.