250 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
ai.robots.txt	6da804e826	chore: add ISSCyberRiskCrawler	2024-09-30 23:50:18 +00:00
Cory Dransfeldt	9c2394f23b	chore: add ISSCyberRiskCrawler	2024-09-30 16:25:20 -07:00
ai.robots.txt	6d9ce1d62a	chore: add sidetrade bot	2024-09-28 20:58:18 +00:00
Cory Dransfeldt	6a988be27f	chore: add sidetrade bot	2024-09-28 13:58:00 -07:00
ai.robots.txt	632e9d6510	Daily update from Dark Visitors	2024-09-28 01:17:19 +00:00
dark-visitors	7851cea4fd	Daily update from Dark Visitors	2024-09-27 01:18:04 +00:00
Glyn Normington	75343c790e	Merge pull request #38 from urvish-p80/main ... Add an additional resource - README.md	2024-09-27 01:26:04 +01:00
ai.robots.txt	44d975c799	Merge pull request #42 from commoncrawl/main ... feat: make CCBot entry more accurate	2024-09-27 00:21:49 +00:00
Glyn Normington	2f67e77ddb	Merge pull request #42 from commoncrawl/main ... feat: make CCBot entry more accurate	2024-09-27 01:21:37 +01:00
Greg Lindahl	a6de89e6bd	feat: make CCBot entry more accurate	2024-09-26 21:41:28 +00:00
Cory Dransfeldt	60bdfa7eb3	Merge pull request #41 from cityrolr/patch-1 ... Update README.md	2024-09-24 12:53:52 -07:00
Julian Mair	af05890b07	Update README.md ... For people who don't use or don't want to use RSS for this, I've added a little explanation of how to subscribe to releases via GitHub.	2024-09-23 23:27:27 +02:00
Urvish Patel	0106d4b15a	Add additional resource - README.md ... A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt.	2024-09-23 08:19:27 -04:00
ai.robots.txt	6b8d7f5890	Daily update from Dark Visitors	2024-09-09 01:16:21 +00:00
dark-visitors	5963cbf9f7	Daily update from Dark Visitors	2024-09-08 01:19:31 +00:00
Glyn Normington	b15b8062ce	Merge pull request #36 from cramforce/patch-1 ... Add instructions for AI bot blocking on Vercel	2024-09-08 01:26:07 +01:00
Malte Ubl	809851ae88	Add instructions for AI bot blocking on Vercel	2024-09-07 15:59:25 -07:00
ai.robots.txt	1c1b423684	chore: add iaskspider/2.0	2024-09-07 02:05:43 +00:00
Cory Dransfeldt	8373294404	chore: add iaskspider/2.0	2024-09-06 19:05:26 -07:00
Cory Dransfeldt	b30ca5f193	Merge pull request #35 from nisbet-hubbard/patch-7 ... Improve main workflow	2024-09-02 18:40:57 -07:00
ai.robots.txt	fb5c995243	Daily update from Dark Visitors	2024-09-03 01:12:57 +00:00
ai.robots.txt	7151f6c569	Removing previously generated files	2024-09-03 01:12:56 +00:00
nisbet-hubbard	cc18b8617c	Update main.yml	2024-09-03 07:48:48 +08:00
ai.robots.txt	c9325c9e18	Daily update from Dark Visitors	2024-09-02 01:15:07 +00:00
ai.robots.txt	567bd00aec	Removing previously generated files	2024-09-02 01:15:07 +00:00
ai.robots.txt	543e993b08	Daily update from Dark Visitors	2024-09-01 01:24:53 +00:00
ai.robots.txt	01589718df	Removing previously generated files	2024-09-01 01:24:52 +00:00
ai.robots.txt	9a7f556d87	Daily update from Dark Visitors	2024-08-31 01:13:04 +00:00
ai.robots.txt	9a4ebb57ee	Removing previously generated files	2024-08-31 01:13:04 +00:00
ai.robots.txt	054c97ad4f	Daily update from Dark Visitors	2024-08-30 01:13:29 +00:00
ai.robots.txt	b2970316d8	Removing previously generated files	2024-08-30 01:13:29 +00:00
ai.robots.txt	008a34ceb4	chore: add ai2bot	2024-08-29 03:07:52 +00:00
ai.robots.txt	3bce634e4a	Removing previously generated files	2024-08-29 03:07:51 +00:00
Cory Dransfeldt	0f8723558f	chore: add ai2bot	2024-08-28 20:07:32 -07:00
ai.robots.txt	6dc900b582	Daily update from Dark Visitors	2024-08-29 01:13:19 +00:00
ai.robots.txt	71eefcdb05	Removing previously generated files	2024-08-29 01:13:19 +00:00
ai.robots.txt	1d417ffab9	Daily update from Dark Visitors	2024-08-28 01:12:35 +00:00
ai.robots.txt	00ef18f93c	Removing previously generated files	2024-08-28 01:12:35 +00:00
ai.robots.txt	84a2376f65	Daily update from Dark Visitors	2024-08-27 01:12:20 +00:00
ai.robots.txt	699862f4bd	Removing previously generated files	2024-08-27 01:12:19 +00:00
ai.robots.txt	ccec3eef15	Daily update from Dark Visitors	2024-08-26 01:11:41 +00:00
ai.robots.txt	6cb9bc8ebf	Removing previously generated files	2024-08-26 01:11:40 +00:00
ai.robots.txt	42a7ca7eda	Daily update from Dark Visitors	2024-08-25 01:16:28 +00:00
ai.robots.txt	907866301f	Removing previously generated files	2024-08-25 01:16:27 +00:00
ai.robots.txt	b202b9e1e3	Daily update from Dark Visitors	2024-08-24 01:09:29 +00:00
ai.robots.txt	ac1250cfa5	Removing previously generated files	2024-08-24 01:09:29 +00:00
ai.robots.txt	d95f2e8072	Daily update from Dark Visitors	2024-08-23 01:10:54 +00:00
ai.robots.txt	61d851baf5	Removing previously generated files	2024-08-23 01:10:53 +00:00
dark-visitors	7bfc1647a8	Daily update from Dark Visitors	2024-08-22 01:11:43 +00:00
ai.robots.txt	3580a7096f	Daily update from Dark Visitors	2024-08-21 01:10:11 +00:00