125 commits

Author	SHA1	Message	Date
ai.robots.txt	5b8650b99b	Update from Dark Visitors	2025-03-29 00:54:10 +00:00
Cory Dransfeldt	ec18af7624	Revert "Merge pull request #91 from deyigifts/perplexity-user" ... This reverts commit 68d1d93714.	2025-03-27 12:51:22 -07:00
ai.robots.txt	68d1d93714	Merge pull request #91 from deyigifts/perplexity-user ... Update perplexity bots	2025-03-27 19:29:30 +00:00
ai.robots.txt	693289bb29	chore: add Brightbot 1.0	2025-02-16 21:37:52 +00:00
ai.robots.txt	89d4c6e5ca	Merge pull request #73 from nisbet-hubbard/patch-8 ... Actually block Semrush’s AI tools	2025-02-01 10:51:01 +00:00
ai.robots.txt	6c552a3daa	Merge pull request #71 from jsheard/patch-1 ... Add Crawlspace	2025-01-20 17:45:42 +00:00
ai.robots.txt	ec454b71d3	Merge pull request #67 from Nightfirecat/semrushbot ... Block SemrushBot	2025-01-06 20:51:56 +00:00
ai.robots.txt	ca8620e28b	Merge pull request #63 from glyn/push-paths ... Convert robots.json more frequently	2025-01-05 05:05:20 +00:00
dark-visitors	609ddca392	Updated from new robots.json	2024-11-24 00:57:06 +00:00
dark-visitors	58985737e7	Updated from new robots.json	2024-11-19 16:46:21 +00:00
dark-visitors	9e06cf3bc9	Updated from new robots.json	2024-10-29 00:52:12 +00:00
fabianegli	0c05461f84	simplify repo and added some tests	2024-10-19 13:06:34 +02:00
ai.robots.txt	a80bd18fb8	Dump out file contents in PHP script	2024-10-19 03:34:29 +00:00
ai.robots.txt	6da804e826	chore: add ISSCyberRiskCrawler	2024-09-30 23:50:18 +00:00
ai.robots.txt	6d9ce1d62a	chore: add sidetrade bot	2024-09-28 20:58:18 +00:00
ai.robots.txt	632e9d6510	Daily update from Dark Visitors	2024-09-28 01:17:19 +00:00
ai.robots.txt	6b8d7f5890	Daily update from Dark Visitors	2024-09-09 01:16:21 +00:00
ai.robots.txt	1c1b423684	chore: add iaskspider/2.0	2024-09-07 02:05:43 +00:00
ai.robots.txt	fb5c995243	Daily update from Dark Visitors	2024-09-03 01:12:57 +00:00
ai.robots.txt	7151f6c569	Removing previously generated files	2024-09-03 01:12:56 +00:00
ai.robots.txt	c9325c9e18	Daily update from Dark Visitors	2024-09-02 01:15:07 +00:00
ai.robots.txt	567bd00aec	Removing previously generated files	2024-09-02 01:15:07 +00:00
ai.robots.txt	543e993b08	Daily update from Dark Visitors	2024-09-01 01:24:53 +00:00
ai.robots.txt	01589718df	Removing previously generated files	2024-09-01 01:24:52 +00:00
ai.robots.txt	9a7f556d87	Daily update from Dark Visitors	2024-08-31 01:13:04 +00:00
ai.robots.txt	9a4ebb57ee	Removing previously generated files	2024-08-31 01:13:04 +00:00
ai.robots.txt	054c97ad4f	Daily update from Dark Visitors	2024-08-30 01:13:29 +00:00
ai.robots.txt	b2970316d8	Removing previously generated files	2024-08-30 01:13:29 +00:00
ai.robots.txt	008a34ceb4	chore: add ai2bot	2024-08-29 03:07:52 +00:00
ai.robots.txt	3bce634e4a	Removing previously generated files	2024-08-29 03:07:51 +00:00
ai.robots.txt	6dc900b582	Daily update from Dark Visitors	2024-08-29 01:13:19 +00:00
ai.robots.txt	71eefcdb05	Removing previously generated files	2024-08-29 01:13:19 +00:00
ai.robots.txt	1d417ffab9	Daily update from Dark Visitors	2024-08-28 01:12:35 +00:00
ai.robots.txt	00ef18f93c	Removing previously generated files	2024-08-28 01:12:35 +00:00
ai.robots.txt	84a2376f65	Daily update from Dark Visitors	2024-08-27 01:12:20 +00:00
ai.robots.txt	699862f4bd	Removing previously generated files	2024-08-27 01:12:19 +00:00
ai.robots.txt	ccec3eef15	Daily update from Dark Visitors	2024-08-26 01:11:41 +00:00
ai.robots.txt	6cb9bc8ebf	Removing previously generated files	2024-08-26 01:11:40 +00:00
ai.robots.txt	42a7ca7eda	Daily update from Dark Visitors	2024-08-25 01:16:28 +00:00
ai.robots.txt	907866301f	Removing previously generated files	2024-08-25 01:16:27 +00:00
ai.robots.txt	b202b9e1e3	Daily update from Dark Visitors	2024-08-24 01:09:29 +00:00
ai.robots.txt	ac1250cfa5	Removing previously generated files	2024-08-24 01:09:29 +00:00
ai.robots.txt	d95f2e8072	Daily update from Dark Visitors	2024-08-23 01:10:54 +00:00
ai.robots.txt	61d851baf5	Removing previously generated files	2024-08-23 01:10:53 +00:00
ai.robots.txt	3580a7096f	Daily update from Dark Visitors	2024-08-21 01:10:11 +00:00
ai.robots.txt	fad335178f	Removing previously generated files	2024-08-21 01:10:10 +00:00
ai.robots.txt	358df0833e	Daily update from Dark Visitors	2024-08-20 01:10:11 +00:00
ai.robots.txt	7e0dd921db	Removing previously generated files	2024-08-20 01:10:11 +00:00
ai.robots.txt	591a99c320	Daily update from Dark Visitors	2024-08-19 01:11:49 +00:00
ai.robots.txt	394e447c78	Removing previously generated files	2024-08-19 01:11:49 +00:00