22 commits

Author	SHA1	Message	Date
Laker Turner	dc15afe847	Update robots.json with Claude respect link	2024-10-07 17:38:01 +01:00
Cory Dransfeldt	9c2394f23b	chore: add ISSCyberRiskCrawler	2024-09-30 16:25:20 -07:00
Cory Dransfeldt	6a988be27f	chore: add sidetrade bot	2024-09-28 13:58:00 -07:00
dark-visitors	7851cea4fd	Daily update from Dark Visitors	2024-09-27 01:18:04 +00:00
Greg Lindahl	a6de89e6bd	feat: make CCBot entry more accurate	2024-09-26 21:41:28 +00:00
dark-visitors	5963cbf9f7	Daily update from Dark Visitors	2024-09-08 01:19:31 +00:00
Cory Dransfeldt	8373294404	chore: add iaskspider/2.0	2024-09-06 19:05:26 -07:00
Cory Dransfeldt	0f8723558f	chore: add ai2bot	2024-08-28 20:07:32 -07:00
dark-visitors	7bfc1647a8	Daily update from Dark Visitors	2024-08-22 01:11:43 +00:00
dark-visitors	5937434aff	Daily update from Dark Visitors	2024-08-15 01:07:15 +00:00
dark-visitors	6a275366be	Daily update from Dark Visitors	2024-08-07 10:50:45 +00:00
Chenghao Mou	944bee0f56	call main after update	2024-08-07 11:31:58 +01:00
dark-visitors	cebf809391	Daily update from Dark Visitors	2024-08-07 00:14:26 +00:00
Chenghao Mou	4cf82b703f	restore original robots.json	2024-08-06 19:50:38 +01:00
dark-visitors	63c7e742c3	Daily update from Dark Visitors	2024-08-06 16:54:29 +00:00
dark-visitors	fdd261dad4	Daily update from Dark Visitors	2024-08-06 16:27:02 +00:00
Joshua Sheard	146fd4ffba	Fix Imagesift user agent	2024-08-04 21:33:04 +01:00
Cory Dransfeldt	1ca936ce11	chore: restore FriendlyCrawler + ImageSift	2024-08-04 12:28:48 -07:00
Mirium999	5826c18909	Add ICC-Crawler	2024-08-04 10:11:25 +09:00
Cory Dransfeldt	b20dfec1e4	chore: drop in additional data	2024-08-01 15:33:07 -07:00
Cory Dransfeldt	efabf3e721	chore: remove test data	2024-08-01 15:25:55 -07:00
Adam Newbold	1fdc79dacb	Adding GitHub Action	2024-08-01 18:17:19 -04:00