Glyn Normington
91a88e2fa8
Merge pull request #113 from rwijnen-um/feature/haproxy
...
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
HAProxy converter added.
2025-04-28 09:00:16 +01:00
Rik Wijnen
a4a9f2ac2b
Tests for HAProxy file added.
2025-04-28 09:30:26 +02:00
Rik Wijnen
66da70905f
Fixed incorrect English sentence.
2025-04-28 09:09:40 +02:00
Rik Wijnen
50e739dd73
HAProxy converter added.
2025-04-28 08:51:02 +02:00
ai.robots.txt
c6c7f1748f
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-26 00:55:12 +00:00
dark-visitors
934ac7b318
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
2025-04-25 00:56:57 +00:00
ai.robots.txt
4654e14e9c
Merge pull request #112 from maiavixen/main
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 07:00:34 +00:00
Glyn Normington
9bf31fbca8
Merge pull request #112 from maiavixen/main
...
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 08:00:24 +01:00
maia
9d846ced45
Update robots.json
...
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
2025-04-24 04:08:20 +02:00
dark-visitors
8d25a424d9
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
2025-04-23 00:56:52 +00:00
ai.robots.txt
bbec639c14
Merge pull request #109 from dennislee1/patch-1
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
AI bots to consider adding
2025-04-22 14:50:26 +00:00
422cf9e29b
Merge pull request #109 from dennislee1/patch-1
...
AI bots to consider adding
2025-04-22 07:50:14 -07:00
Dennis Lee
33c5ce1326
Update robots.json
...
Updated robots list with five new proposed AI bots:
aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
2025-04-21 18:55:11 +01:00
774b1ddf52
Merge pull request #107 from glyn/sponsorship
...
/ run-tests (push) Has been cancelled
Clarify our position on sponsorship
2025-04-18 11:40:06 -07:00
Glyn Normington
b1856e6988
Donations
2025-04-18 18:40:44 +01:00
Glyn Normington
d05ede8fe1
Clarify our position on sponsorship
...
Some firms, including those with .ai domains, have
offered to sponsor this project. So make our position
clear.
2025-04-18 17:46:56 +01:00
ai.robots.txt
e0cdb278fb
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-16 00:57:11 +00:00
dark-visitors
a96e330989
Update from Dark Visitors
/ run-tests (push) Waiting to run
2025-04-15 00:57:01 +00:00
156e6baa09
Merge pull request #105 from jsheard/patch-1
...
/ run-tests (push) Waiting to run
Include "AI Agents" from Dark Visitors
2025-04-14 10:08:38 -07:00
Joshua Sheard
d9f882a9b2
Include "AI Agents" from Dark Visitors
2025-04-14 15:46:01 +01:00
dark-visitors
305188b2e7
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-11 00:55:52 +00:00
ai.robots.txt
4a764bba18
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
/ run-tests (push) Waiting to run
chore(robots.json): adds imgproxy crawler
2025-04-10 19:22:34 +00:00
a891ad7213
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
chore(robots.json): adds imgproxy crawler
2025-04-10 12:22:23 -07:00
b65f45e408
chore(robots.json): adds imgproxy crawler
2025-04-10 10:12:51 -07:00
Glyn Normington
49e58b1573
Merge pull request #100 from fbartho/fb/fix-perplexity-users
...
/ run-tests (push) Has been cancelled
Fix html-mangled hyphen in 'Perplexity-Users' bot name
2025-04-05 17:32:19 +01:00
Frederic Barthelemy
c6f308cbd0
PR Feedback: log special-case, comment consistency
2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
fix python complaining about f-string syntax
...
```
python code/tests.py
Traceback (most recent call last):
File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
return f"({"|".join(map(re.escape, lst))})"
^
SyntaxError: f-string: expecting '}'
```
2025-04-04 15:20:30 -07:00
ai.robots.txt
5b8650b99b
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-03-29 00:54:10 +00:00
dark-visitors
c249de99a3
Update from Dark Visitors
2025-03-28 00:54:28 +00:00
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user"
...
This reverts commit 68d1d93714
2025-03-27 12:51:22 -07:00
ai.robots.txt
6851413c52
Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export
...
Implement Nginx configuration snippet export
2025-03-27 19:49:15 +00:00
Glyn Normington
dba03d809c
Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export
...
Implement Nginx configuration snippet export
2025-03-27 19:49:05 +00:00
ai.robots.txt
68d1d93714
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 19:29:30 +00:00
1183187be9
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 12:29:21 -07:00
Thomas Leister
7c3b5a2cb2
Add tests for Nginx config generator
2025-03-27 18:28:21 +01:00
Thomas Leister
4f3f4cd0dd
Add assembled version of nginx-block-ai-bots.conf file
2025-03-27 12:43:36 +01:00
Thomas Leister
5a312c5f4d
Mention Nginx config feature in README
2025-03-27 12:43:29 +01:00
Thomas Leister
da85207314
Implement new function "json_to_nginx" which outputs an Nginx
...
configuration snippet
2025-03-27 12:27:09 +01:00
deyigifts
6ecfcdfcbf
Update perplexity bot
...
Update based on perplexity bot docs
2025-03-24 14:16:57 +08:00
5e7c3c432f
Merge pull request #83 from glyn/81-doc-testing
...
Document testing in README
2025-02-19 09:19:44 -08:00
Glyn Normington
9f41d4c11c
Merge pull request #84 from sideeffect42/tests-workflow
...
Add run-tests workflow
2025-02-18 19:42:55 +00:00
Dennis Camera
8a74896333
Add workflow to run tests on pull request or push to main
2025-02-18 20:30:27 +01:00
Glyn Normington
1d55a205e4
Document testing in README
...
Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81
2025-02-18 16:49:08 +00:00
Glyn Normington
8494a7fcaa
Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt
...
.htaccess: Allow robots access to `/robots.txt`
2025-02-18 16:42:36 +00:00
Dennis Camera
c7c1e7b96f
robots.py: Make executable
2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3
Update tests and convert to stock unittest
...
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.
Added some more test cases with "special" characters to test the escaping code
better.
2025-02-18 12:55:15 +01:00
Dennis Camera
0bd3fa63b8
table-of-bot-metrics.md: Escape robot names for Markdown table
...
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.
The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9
.htaccess: Make regex in RewriteCond safe
...
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87
.htaccess: Allow robots access to /robots.txt
2025-02-18 12:49:29 +01:00
