diff --git a/src/meta/robots.liquid b/src/meta/robots.liquid index f2f2c3c2..5cec9f3f 100644 --- a/src/meta/robots.liquid +++ b/src/meta/robots.liquid @@ -7,7 +7,8 @@ Sitemap: https://coryd.dev/sitemap.xml User-agent: * Disallow: -{% for robot in robots %} + +{% for robot in robots -%} User-agent: {{ robot }} -Disallow: / -{% endfor %} \ No newline at end of file +{% endfor -%} +Disallow: / \ No newline at end of file diff --git a/src/posts/2024/go-ahead-and-block-ai-web-crawlers.md b/src/posts/2024/go-ahead-and-block-ai-web-crawlers.md index 601f519d..883912c6 100644 --- a/src/posts/2024/go-ahead-and-block-ai-web-crawlers.md +++ b/src/posts/2024/go-ahead-and-block-ai-web-crawlers.md @@ -68,6 +68,6 @@ Disallow: / - [I’m blocking AI-crawlers](https://roelant.net/en/2023/im-blocking-ai-crawlers/) - [Block the Bots that Feed “AI” Models by Scraping Your Website](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/) -**Update March 27, 2024:** Many thanks to Jens for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement. +**Update March 27, 2024:** Many thanks to [Jens](https://meiert.com/en/) for pointing out that the `User-agent` rules can be safely combined preceding a `Disallow` statement. [^1]: I've yet to definitively identify Arc Search's user agent but I'd like to, so I can block it and share it — but that assumes they respect `robots.txt` declarations. \ No newline at end of file