Glyn Normington
fb8188c49d
Merge pull request #115 from glyn/syntax
...
Fix Python syntax error
2025-05-01 11:28:54 +01:00
Glyn Normington
ec995cd686
Fix Python syntax error
2025-05-01 11:27:40 +01:00
Crazyroostereye
1310dbae46
Added a Caddyfile converter ( #110 )
...
Co-authored-by: Julian Beittel <julian@beittel.net>
Co-authored-by: Glyn Normington <work@underlap.org>
2025-05-01 11:21:32 +01:00
Glyn Normington
91a88e2fa8
Merge pull request #113 from rwijnen-um/feature/haproxy
...
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
HAProxy converter added.
2025-04-28 09:00:16 +01:00
Rik Wijnen
a4a9f2ac2b
Tests for HAProxy file added.
2025-04-28 09:30:26 +02:00
Rik Wijnen
66da70905f
Fixed incorrect English sentence.
2025-04-28 09:09:40 +02:00
Rik Wijnen
50e739dd73
HAProxy converter added.
2025-04-28 08:51:02 +02:00
ai.robots.txt
c6c7f1748f
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-26 00:55:12 +00:00
dark-visitors
934ac7b318
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
2025-04-25 00:56:57 +00:00
ai.robots.txt
4654e14e9c
Merge pull request #112 from maiavixen/main
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 07:00:34 +00:00
Glyn Normington
9bf31fbca8
Merge pull request #112 from maiavixen/main
...
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 08:00:24 +01:00
maia
9d846ced45
Update robots.json
...
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
2025-04-24 04:08:20 +02:00
dark-visitors
8d25a424d9
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
2025-04-23 00:56:52 +00:00
ai.robots.txt
bbec639c14
Merge pull request #109 from dennislee1/patch-1
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
AI bots to consider adding
2025-04-22 14:50:26 +00:00
422cf9e29b
Merge pull request #109 from dennislee1/patch-1
...
AI bots to consider adding
2025-04-22 07:50:14 -07:00
Dennis Lee
33c5ce1326
Update robots.json
...
Updated robots list with five new proposed AI bots:
aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
2025-04-21 18:55:11 +01:00
774b1ddf52
Merge pull request #107 from glyn/sponsorship
...
/ run-tests (push) Has been cancelled
Clarify our position on sponsorship
2025-04-18 11:40:06 -07:00
Glyn Normington
b1856e6988
Donations
2025-04-18 18:40:44 +01:00
Glyn Normington
d05ede8fe1
Clarify our position on sponsorship
...
Some firms, including those with .ai domains, have
offered to sponsor this project. So make our position
clear.
2025-04-18 17:46:56 +01:00
ai.robots.txt
e0cdb278fb
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-16 00:57:11 +00:00
dark-visitors
a96e330989
Update from Dark Visitors
/ run-tests (push) Waiting to run
2025-04-15 00:57:01 +00:00
156e6baa09
Merge pull request #105 from jsheard/patch-1
...
/ run-tests (push) Waiting to run
Include "AI Agents" from Dark Visitors
2025-04-14 10:08:38 -07:00
Joshua Sheard
d9f882a9b2
Include "AI Agents" from Dark Visitors
2025-04-14 15:46:01 +01:00
dark-visitors
305188b2e7
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-11 00:55:52 +00:00
ai.robots.txt
4a764bba18
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
/ run-tests (push) Waiting to run
chore(robots.json): adds imgproxy crawler
2025-04-10 19:22:34 +00:00
a891ad7213
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
chore(robots.json): adds imgproxy crawler
2025-04-10 12:22:23 -07:00
b65f45e408
chore(robots.json): adds imgproxy crawler
2025-04-10 10:12:51 -07:00
Glyn Normington
49e58b1573
Merge pull request #100 from fbartho/fb/fix-perplexity-users
...
/ run-tests (push) Has been cancelled
Fix html-mangled hyphen in 'Perplexity-Users' bot name
2025-04-05 17:32:19 +01:00
Frederic Barthelemy
c6f308cbd0
PR Feedback: log special-case, comment consistency
2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
fix python complaining about f-string syntax
...
```
python code/tests.py
Traceback (most recent call last):
File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
return f"({"|".join(map(re.escape, lst))})"
^
SyntaxError: f-string: expecting '}'
```
2025-04-04 15:20:30 -07:00
ai.robots.txt
5b8650b99b
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-03-29 00:54:10 +00:00
dark-visitors
c249de99a3
Update from Dark Visitors
2025-03-28 00:54:28 +00:00
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user"
...
This reverts commit 68d1d93714
2025-03-27 12:51:22 -07:00
ai.robots.txt
6851413c52
Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export
...
Implement Nginx configuration snippet export
2025-03-27 19:49:15 +00:00
Glyn Normington
dba03d809c
Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export
...
Implement Nginx configuration snippet export
2025-03-27 19:49:05 +00:00
ai.robots.txt
68d1d93714
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 19:29:30 +00:00
1183187be9
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 12:29:21 -07:00
Thomas Leister
7c3b5a2cb2
Add tests for Nginx config generator
2025-03-27 18:28:21 +01:00
Thomas Leister
4f3f4cd0dd
Add assembled version of nginx-block-ai-bots.conf file
2025-03-27 12:43:36 +01:00
Thomas Leister
5a312c5f4d
Mention Nginx config feature in README
2025-03-27 12:43:29 +01:00
Thomas Leister
da85207314
Implement new function "json_to_nginx" which outputs an Nginx
...
configuration snippet
2025-03-27 12:27:09 +01:00
deyigifts
6ecfcdfcbf
Update perplexity bot
...
Update based on perplexity bot docs
2025-03-24 14:16:57 +08:00
5e7c3c432f
Merge pull request #83 from glyn/81-doc-testing
...
Document testing in README
2025-02-19 09:19:44 -08:00
Glyn Normington
9f41d4c11c
Merge pull request #84 from sideeffect42/tests-workflow
...
Add run-tests workflow
2025-02-18 19:42:55 +00:00
Dennis Camera
8a74896333
Add workflow to run tests on pull request or push to main
2025-02-18 20:30:27 +01:00
Glyn Normington
1d55a205e4
Document testing in README
...
Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81
2025-02-18 16:49:08 +00:00
Glyn Normington
8494a7fcaa
Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt
...
.htaccess: Allow robots access to `/robots.txt`
2025-02-18 16:42:36 +00:00
Dennis Camera
c7c1e7b96f
robots.py: Make executable
2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3
Update tests and convert to stock unittest
...
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.
Added some more test cases with "special" characters to test the escaping code
better.
2025-02-18 12:55:15 +01:00
