cdransf/coryd.dev-eleventy
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: robots formatting

Browse source
This commit is contained in:
Cory Dransfeldt 2024-03-27 09:08:50 -07:00
parent f5b3ebbacb
commit c8c6c0e731
No known key found for this signature in database
2 changed files with 5 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
src/meta/robots.liquid
View file

@ -7,7 +7,8 @@ Sitemap: https://coryd.dev/sitemap.xml
User-agent: *
Disallow:
{% for robot in robots %}
{% for robot in robots -%}
User-agent: {{ robot }}
Disallow: /
{% endfor %}
{% endfor -%}
Disallow: /

2
src/posts/2024/go-ahead-and-block-ai-web-crawlers.md
View file

@ -68,6 +68,6 @@ Disallow: /
- [I’m blocking AI-crawlers](https://roelant.net/en/2023/im-blocking-ai-crawlers/)
- [Block the Bots that Feed “AI” Models by Scraping Your Website](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/)
**Update March 27, 2024:** Many thanks to Jens for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement.
**Update March 27, 2024:** Many thanks to [Jens](https://meiert.com/en/) for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement.
[^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations.