cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-17 16:03:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
415 commits 4 branches 31 tags 971 KiB
Commit graph

145 commits

Author SHA1 Message Date
ai.robots.txt
dd1ed174b7 Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details 
chore(robots.json): adds Google-CloudVertexBot
 2025-05-16 11:35:15 +00:00
ai.robots.txt
1c470babbe Merge pull request #123 from joehoyle/patch-1 
Fix JSON syntax error
 2025-05-15 16:12:30 +00:00
ai.robots.txt
c6c7f1748f Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-04-26 00:55:12 +00:00
ai.robots.txt
4654e14e9c Merge pull request #112 from maiavixen/main
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details 
Fixed meta-external* being titlecase, and removed period for consistency
 2025-04-24 07:00:34 +00:00
ai.robots.txt
bbec639c14 Merge pull request #109 from dennislee1/patch-1
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details 
AI bots to consider adding
 2025-04-22 14:50:26 +00:00
ai.robots.txt
e0cdb278fb Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-04-16 00:57:11 +00:00
ai.robots.txt
4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot
Some checks are pending
/ run-tests (push) Waiting to run
Details 
chore(robots.json): adds imgproxy crawler
 2025-04-10 19:22:34 +00:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
ai.robots.txt
5b8650b99b Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-03-29 00:54:10 +00:00
Cory Dransfeldt
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00
Dennis Camera
0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table 
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.

The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
 2025-02-18 12:53:27 +01:00
ai.robots.txt
693289bb29 chore: add Brightbot 1.0 2025-02-16 21:37:52 +00:00
ai.robots.txt
89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 
Actually block Semrush’s AI tools
 2025-02-01 10:51:01 +00:00
ai.robots.txt
6c552a3daa Merge pull request #71 from jsheard/patch-1 
Add Crawlspace
 2025-01-20 17:45:42 +00:00
ai.robots.txt
ec454b71d3 Merge pull request #67 from Nightfirecat/semrushbot 
Block SemrushBot
 2025-01-06 20:51:56 +00:00
ai.robots.txt
ca8620e28b Merge pull request #63 from glyn/push-paths 
Convert robots.json more frequently
 2025-01-05 05:05:20 +00:00
dark-visitors
609ddca392 Updated from new robots.json 2024-11-24 00:57:06 +00:00
dark-visitors
58985737e7 Updated from new robots.json 2024-11-19 16:46:21 +00:00
dark-visitors
9e06cf3bc9 Updated from new robots.json 2024-10-29 00:52:12 +00:00
fabianegli
0c05461f84 simplify repo and added some tests 2024-10-19 13:06:34 +02:00
ai.robots.txt
6a359e7fd7 Fix typo and trigger rerun of main job 2024-10-19 03:43:00 +00:00
ai.robots.txt
a80bd18fb8 Dump out file contents in PHP script 2024-10-19 03:34:29 +00:00
ai.robots.txt
9be286626d Merge pull request #43 from lxjv/main 
Update robots.json with Claude respect link
 2024-10-08 02:30:17 +00:00
ai.robots.txt
6da804e826 chore: add ISSCyberRiskCrawler 2024-09-30 23:50:18 +00:00
ai.robots.txt
6d9ce1d62a chore: add sidetrade bot 2024-09-28 20:58:18 +00:00
ai.robots.txt
632e9d6510 Daily update from Dark Visitors 2024-09-28 01:17:19 +00:00
ai.robots.txt
44d975c799 Merge pull request #42 from commoncrawl/main 
feat: make CCBot entry more accurate
 2024-09-27 00:21:49 +00:00
ai.robots.txt
6b8d7f5890 Daily update from Dark Visitors 2024-09-09 01:16:21 +00:00
ai.robots.txt
1c1b423684 chore: add iaskspider/2.0 2024-09-07 02:05:43 +00:00
ai.robots.txt
fb5c995243 Daily update from Dark Visitors 2024-09-03 01:12:57 +00:00
ai.robots.txt
7151f6c569 Removing previously generated files 2024-09-03 01:12:56 +00:00
ai.robots.txt
c9325c9e18 Daily update from Dark Visitors 2024-09-02 01:15:07 +00:00
ai.robots.txt
567bd00aec Removing previously generated files 2024-09-02 01:15:07 +00:00
ai.robots.txt
543e993b08 Daily update from Dark Visitors 2024-09-01 01:24:53 +00:00
ai.robots.txt
01589718df Removing previously generated files 2024-09-01 01:24:52 +00:00
ai.robots.txt
9a7f556d87 Daily update from Dark Visitors 2024-08-31 01:13:04 +00:00
ai.robots.txt
9a4ebb57ee Removing previously generated files 2024-08-31 01:13:04 +00:00
ai.robots.txt
054c97ad4f Daily update from Dark Visitors 2024-08-30 01:13:29 +00:00
ai.robots.txt
b2970316d8 Removing previously generated files 2024-08-30 01:13:29 +00:00
ai.robots.txt
008a34ceb4 chore: add ai2bot 2024-08-29 03:07:52 +00:00
ai.robots.txt
3bce634e4a Removing previously generated files 2024-08-29 03:07:51 +00:00
ai.robots.txt
6dc900b582 Daily update from Dark Visitors 2024-08-29 01:13:19 +00:00
ai.robots.txt
71eefcdb05 Removing previously generated files 2024-08-29 01:13:19 +00:00
ai.robots.txt
1d417ffab9 Daily update from Dark Visitors 2024-08-28 01:12:35 +00:00
ai.robots.txt
00ef18f93c Removing previously generated files 2024-08-28 01:12:35 +00:00
ai.robots.txt
84a2376f65 Daily update from Dark Visitors 2024-08-27 01:12:20 +00:00
ai.robots.txt
699862f4bd Removing previously generated files 2024-08-27 01:12:19 +00:00
ai.robots.txt
ccec3eef15 Daily update from Dark Visitors 2024-08-26 01:11:41 +00:00
ai.robots.txt
6cb9bc8ebf Removing previously generated files 2024-08-26 01:11:40 +00:00