From be9d22abedf791936f9d8ea144db2c632252e872 Mon Sep 17 00:00:00 2001 From: Cory Dransfeldt Date: Wed, 27 Mar 2024 10:58:49 -0700 Subject: [PATCH] Create README.md --- README.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..dc465e1 --- /dev/null +++ b/README.md @@ -0,0 +1,5 @@ +# AI robots.txt + +This is an open list of web crawlers associated with AI companies and the training of LLMs to block. We encourage you to contribute to and implement this list on your own site. + +A number of these crawlers have been sourced from [Dark Visitors](https://darkvisitors.com) and we appreciate the ongoing effort they put in to track these crawlers.