cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-03 19:03:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
367 commits 2 branches 27 tags 521 KiB
Commit graph

137 commits

Author SHA1 Message Date
ai.robots.txt
5b8650b99b Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-03-29 00:54:10 +00:00
Cory Dransfeldt
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00
Dennis Camera
0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table 
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.

The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
 2025-02-18 12:53:27 +01:00
ai.robots.txt
693289bb29 chore: add Brightbot 1.0 2025-02-16 21:37:52 +00:00
ai.robots.txt
89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 
Actually block Semrush’s AI tools
 2025-02-01 10:51:01 +00:00
ai.robots.txt
6c552a3daa Merge pull request #71 from jsheard/patch-1 
Add Crawlspace
 2025-01-20 17:45:42 +00:00
ai.robots.txt
ec454b71d3 Merge pull request #67 from Nightfirecat/semrushbot 
Block SemrushBot
 2025-01-06 20:51:56 +00:00
ai.robots.txt
ca8620e28b Merge pull request #63 from glyn/push-paths 
Convert robots.json more frequently
 2025-01-05 05:05:20 +00:00
dark-visitors
609ddca392 Updated from new robots.json 2024-11-24 00:57:06 +00:00
dark-visitors
58985737e7 Updated from new robots.json 2024-11-19 16:46:21 +00:00
dark-visitors
9e06cf3bc9 Updated from new robots.json 2024-10-29 00:52:12 +00:00
fabianegli
0c05461f84 simplify repo and added some tests 2024-10-19 13:06:34 +02:00
ai.robots.txt
6a359e7fd7 Fix typo and trigger rerun of main job 2024-10-19 03:43:00 +00:00
ai.robots.txt
a80bd18fb8 Dump out file contents in PHP script 2024-10-19 03:34:29 +00:00
ai.robots.txt
9be286626d Merge pull request #43 from lxjv/main 
Update robots.json with Claude respect link
 2024-10-08 02:30:17 +00:00
ai.robots.txt
6da804e826 chore: add ISSCyberRiskCrawler 2024-09-30 23:50:18 +00:00
ai.robots.txt
6d9ce1d62a chore: add sidetrade bot 2024-09-28 20:58:18 +00:00
ai.robots.txt
632e9d6510 Daily update from Dark Visitors 2024-09-28 01:17:19 +00:00
ai.robots.txt
44d975c799 Merge pull request #42 from commoncrawl/main 
feat: make CCBot entry more accurate
 2024-09-27 00:21:49 +00:00
ai.robots.txt
6b8d7f5890 Daily update from Dark Visitors 2024-09-09 01:16:21 +00:00
ai.robots.txt
1c1b423684 chore: add iaskspider/2.0 2024-09-07 02:05:43 +00:00
ai.robots.txt
fb5c995243 Daily update from Dark Visitors 2024-09-03 01:12:57 +00:00
ai.robots.txt
7151f6c569 Removing previously generated files 2024-09-03 01:12:56 +00:00
ai.robots.txt
c9325c9e18 Daily update from Dark Visitors 2024-09-02 01:15:07 +00:00
ai.robots.txt
567bd00aec Removing previously generated files 2024-09-02 01:15:07 +00:00
ai.robots.txt
543e993b08 Daily update from Dark Visitors 2024-09-01 01:24:53 +00:00
ai.robots.txt
01589718df Removing previously generated files 2024-09-01 01:24:52 +00:00
ai.robots.txt
9a7f556d87 Daily update from Dark Visitors 2024-08-31 01:13:04 +00:00
ai.robots.txt
9a4ebb57ee Removing previously generated files 2024-08-31 01:13:04 +00:00
ai.robots.txt
054c97ad4f Daily update from Dark Visitors 2024-08-30 01:13:29 +00:00
ai.robots.txt
b2970316d8 Removing previously generated files 2024-08-30 01:13:29 +00:00
ai.robots.txt
008a34ceb4 chore: add ai2bot 2024-08-29 03:07:52 +00:00
ai.robots.txt
3bce634e4a Removing previously generated files 2024-08-29 03:07:51 +00:00
ai.robots.txt
6dc900b582 Daily update from Dark Visitors 2024-08-29 01:13:19 +00:00
ai.robots.txt
71eefcdb05 Removing previously generated files 2024-08-29 01:13:19 +00:00
ai.robots.txt
1d417ffab9 Daily update from Dark Visitors 2024-08-28 01:12:35 +00:00
ai.robots.txt
00ef18f93c Removing previously generated files 2024-08-28 01:12:35 +00:00
ai.robots.txt
84a2376f65 Daily update from Dark Visitors 2024-08-27 01:12:20 +00:00
ai.robots.txt
699862f4bd Removing previously generated files 2024-08-27 01:12:19 +00:00
ai.robots.txt
ccec3eef15 Daily update from Dark Visitors 2024-08-26 01:11:41 +00:00
ai.robots.txt
6cb9bc8ebf Removing previously generated files 2024-08-26 01:11:40 +00:00
ai.robots.txt
42a7ca7eda Daily update from Dark Visitors 2024-08-25 01:16:28 +00:00
ai.robots.txt
907866301f Removing previously generated files 2024-08-25 01:16:27 +00:00
ai.robots.txt
b202b9e1e3 Daily update from Dark Visitors 2024-08-24 01:09:29 +00:00
ai.robots.txt
ac1250cfa5 Removing previously generated files 2024-08-24 01:09:29 +00:00
ai.robots.txt
d95f2e8072 Daily update from Dark Visitors 2024-08-23 01:10:54 +00:00
ai.robots.txt
61d851baf5 Removing previously generated files 2024-08-23 01:10:53 +00:00
ai.robots.txt
3580a7096f Daily update from Dark Visitors 2024-08-21 01:10:11 +00:00
ai.robots.txt
fad335178f Removing previously generated files 2024-08-21 01:10:10 +00:00