cdransf/coryd.dev-eleventy
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: update robots + posts

Browse source
This commit is contained in:
Cory Dransfeldt 2024-03-02 11:23:48 -08:00
parent 766d70c59e
commit b5cc034f1e
No known key found for this signature in database
2 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
src/_data/json/robots.js
View file

@ -14,6 +14,7 @@ export default [
'FacebookBot',
'Google-Extended',
'GPTBot',
'ImagesiftBot',
'magpie-crawler',
'omgili',
'omgilibot',

7
src/posts/2024/go-ahead-and-block-ai-web-crawlers.md
View file

@ -83,6 +83,9 @@ Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: magpie-crawler
Disallow: /
@ -105,4 +108,8 @@ User-agent: YouBot
Disallow: /
```
**Other great posts on the subject:**
- [I’m blocking AI-crawlers](https://roelant.net/en/2023/im-blocking-ai-crawlers/)
- [Block the Bots that Feed “AI” Models by Scraping Your Website](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/)
[^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations.