From 0106d4b15a4e1d3e0fe15ff0c137b53767afe7b4 Mon Sep 17 00:00:00 2001 From: Urvish Patel <169079981+urvish-p80@users.noreply.github.com> Date: Mon, 23 Sep 2024 08:19:27 -0400 Subject: [PATCH] Add additional resource - README.md A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c2d79f5..9502b0d 100644 --- a/README.md +++ b/README.md @@ -33,3 +33,4 @@ If you use [Cloudflare's hard block](https://blog.cloudflare.com/declaring-your- - [Blocking Bots With 11ty And Apache](https://flamedfury.com/posts/blocking-bots-with-11ty-and-apache/) by fLaMEd fury - [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman - [Blocking AI web crawlers](https://underlap.org/blocking-ai-web-crawlers) by Glyn Normington +- [Block AI Bots from Crawling Websites Using Robots.txt](https://originality.ai/ai-bot-blocking) by Jonathan Gillham, Originality.AI