cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-17 16:03:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
413 commits 4 branches 31 tags 971 KiB
Commit graph

16 commits

Author SHA1 Message Date
ai.robots.txt
1c470babbe Merge pull request #123 from joehoyle/patch-1 
Fix JSON syntax error
 2025-05-15 16:12:30 +00:00
ai.robots.txt
c6c7f1748f Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-04-26 00:55:12 +00:00
ai.robots.txt
4654e14e9c Merge pull request #112 from maiavixen/main
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details 
Fixed meta-external* being titlecase, and removed period for consistency
 2025-04-24 07:00:34 +00:00
ai.robots.txt
bbec639c14 Merge pull request #109 from dennislee1/patch-1
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details 
AI bots to consider adding
 2025-04-22 14:50:26 +00:00
ai.robots.txt
e0cdb278fb Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-04-16 00:57:11 +00:00
ai.robots.txt
4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot
Some checks are pending
/ run-tests (push) Waiting to run
Details 
chore(robots.json): adds imgproxy crawler
 2025-04-10 19:22:34 +00:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
ai.robots.txt
5b8650b99b Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-03-29 00:54:10 +00:00
Cory Dransfeldt
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00
Dennis Camera
a884a2afb9 .htaccess: Make regex in RewriteCond safe 
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
 2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87 .htaccess: Allow robots access to /robots.txt 2025-02-18 12:49:29 +01:00
ai.robots.txt
693289bb29 chore: add Brightbot 1.0 2025-02-16 21:37:52 +00:00
ai.robots.txt
89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 
Actually block Semrush’s AI tools
 2025-02-01 10:51:01 +00:00
ai.robots.txt
6c552a3daa Merge pull request #71 from jsheard/patch-1 
Add Crawlspace
 2025-01-20 17:45:42 +00:00
Massimo Gismondi
933aa6159d Implementing htaccess generation 2025-01-07 11:02:29 +01:00