Frederic Barthelemy
c6f308cbd0
PR Feedback: log special-case, comment consistency
2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
fix python complaining about f-string syntax
...
```
python code/tests.py
Traceback (most recent call last):
File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
return f"({"|".join(map(re.escape, lst))})"
^
SyntaxError: f-string: expecting '}'
```
2025-04-04 15:20:30 -07:00
Thomas Leister
7c3b5a2cb2
Add tests for Nginx config generator
2025-03-27 18:28:21 +01:00
Thomas Leister
da85207314
Implement new function "json_to_nginx" which outputs an Nginx
...
configuration snippet
2025-03-27 12:27:09 +01:00
Dennis Camera
c7c1e7b96f
robots.py: Make executable
2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3
Update tests and convert to stock unittest
...
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.
Added some more test cases with "special" characters to test the escaping code
better.
2025-02-18 12:55:15 +01:00
Dennis Camera
0bd3fa63b8
table-of-bot-metrics.md: Escape robot names for Markdown table
...
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.
The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9
.htaccess: Make regex in RewriteCond safe
...
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87
.htaccess: Allow robots access to /robots.txt
2025-02-18 12:49:29 +01:00
Massimo Gismondi
4f03818280
Removed if condition and added a little comments
2025-01-20 06:51:06 +01:00
Massimo Gismondi
8aee2f24bb
Fixed space in comment
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:39:07 +01:00
Massimo Gismondi
b455af66e7
Adding clarification about performance and code comment
2025-01-17 21:42:08 +01:00
Massimo Gismondi
933aa6159d
Implementing htaccess generation
2025-01-07 11:02:29 +01:00
Glyn Normington
e4c12ee2f8
Rename in test code
2025-01-04 05:03:48 +00:00
Glyn Normington
3a43714908
Rename Python code
...
The name dark_visitors.py gives the impression that the code is entirely
related to the dark visitors website, whereas the update command relates
to dark visitors and the convert command is unrelated to dark visitors.
2025-01-04 04:55:34 +00:00
fabianegli
3ab22bc498
make conversions and updates separately triggerable
2024-10-19 19:56:41 +02:00
fabianegli
6ab8fb2d37
no more failure when run without network
2024-10-19 19:11:01 +02:00
fabianegli
0c05461f84
simplify repo and added some tests
2024-10-19 13:06:34 +02:00
Glyn Normington
38a388097c
Fix typo and trigger rerun of main job
2024-10-19 04:42:27 +01:00
Glyn Normington
bdf30be7dc
Dump out file contents in PHP script
2024-10-19 04:33:46 +01:00
Ivan Sagalaev
7bb5efd462
Sort the content case-insensitively before dumping to JSON
2024-10-17 21:08:43 -04:00
407b9e12e6
chore: sort output
2024-08-14 17:10:29 -07:00
Chenghao Mou
944bee0f56
call main after update
2024-08-07 11:31:58 +01:00
Chenghao Mou
2a3685385c
restrict scope
2024-08-06 19:33:49 +01:00
Chenghao Mou
55e92f4324
update existing ones
2024-08-06 17:48:06 +01:00
Chenghao Mou
8ab1e30a6c
test workflow
2024-08-06 17:12:26 +01:00
Chenghao Mou
192bf67631
add dark visitor workflow
2024-08-06 17:02:23 +01:00
Adam Newbold
1fdc79dacb
Adding GitHub Action
2024-08-01 18:17:19 -04:00
