cdransf/coryd.dev-eleventy
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: gh banner

Browse source
This commit is contained in:
Cory Dransfeldt 2024-03-31 07:47:37 -07:00
parent 9070daffcd
commit 5abc4f8b6b
No known key found for this signature in database
3 changed files with 17 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/posts/2024/go-ahead-and-block-ai-web-crawlers.md
View file

@ -71,4 +71,6 @@ Disallow: /
**Update March 27, 2024:** Many thanks to [Jens](https://meiert.com/en/) for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement.
{% render "partials/banners/github.liquid", url: 'https://github.com/ai-robots-txt/ai.robots.txt' %}
[^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations.