Crazyroostereye
1310dbae46
Added a Caddyfile converter ( #110 )
...
Co-authored-by: Julian Beittel <julian@beittel.net>
Co-authored-by: Glyn Normington <work@underlap.org>
2025-05-01 11:21:32 +01:00
Rik Wijnen
50e739dd73
HAProxy converter added.
2025-04-28 08:51:02 +02:00
Joshua Sheard
d9f882a9b2
Include "AI Agents" from Dark Visitors
2025-04-14 15:46:01 +01:00
Frederic Barthelemy
c6f308cbd0
PR Feedback: log special-case, comment consistency
2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
fix python complaining about f-string syntax
...
```
python code/tests.py
Traceback (most recent call last):
File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
return f"({"|".join(map(re.escape, lst))})"
^
SyntaxError: f-string: expecting '}'
```
2025-04-04 15:20:30 -07:00
Thomas Leister
da85207314
Implement new function "json_to_nginx" which outputs an Nginx
...
configuration snippet
2025-03-27 12:27:09 +01:00
Dennis Camera
c7c1e7b96f
robots.py: Make executable
2025-02-18 12:55:17 +01:00
Dennis Camera
0bd3fa63b8
table-of-bot-metrics.md: Escape robot names for Markdown table
...
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.
The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9
.htaccess: Make regex in RewriteCond safe
...
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87
.htaccess: Allow robots access to /robots.txt
2025-02-18 12:49:29 +01:00
Massimo Gismondi
4f03818280
Removed if condition and added a little comments
2025-01-20 06:51:06 +01:00
Massimo Gismondi
8aee2f24bb
Fixed space in comment
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:39:07 +01:00
Massimo Gismondi
b455af66e7
Adding clarification about performance and code comment
2025-01-17 21:42:08 +01:00
Massimo Gismondi
933aa6159d
Implementing htaccess generation
2025-01-07 11:02:29 +01:00
Glyn Normington
3a43714908
Rename Python code
...
The name dark_visitors.py gives the impression that the code is entirely
related to the dark visitors website, whereas the update command relates
to dark visitors and the convert command is unrelated to dark visitors.
2025-01-04 04:55:34 +00:00
