Commit graph

26 commits

Author SHA1 Message Date
Thomas Leister
7c3b5a2cb2
Add tests for Nginx config generator 2025-03-27 18:28:21 +01:00
Thomas Leister
da85207314
Implement new function "json_to_nginx" which outputs an Nginx
configuration snippet
2025-03-27 12:27:09 +01:00
Dennis Camera
c7c1e7b96f robots.py: Make executable 2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3 Update tests and convert to stock unittest
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.

Added some more test cases with "special" characters to test the escaping code
better.
2025-02-18 12:55:15 +01:00
Dennis Camera
0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.

The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9 .htaccess: Make regex in RewriteCond safe
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87 .htaccess: Allow robots access to /robots.txt 2025-02-18 12:49:29 +01:00
Massimo Gismondi
4f03818280 Removed if condition and added a little comments 2025-01-20 06:51:06 +01:00
Massimo Gismondi
8aee2f24bb
Fixed space in comment
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:39:07 +01:00
Massimo Gismondi
b455af66e7 Adding clarification about performance and code comment 2025-01-17 21:42:08 +01:00
Massimo Gismondi
933aa6159d Implementing htaccess generation 2025-01-07 11:02:29 +01:00
Glyn Normington
e4c12ee2f8 Rename in test code 2025-01-04 05:03:48 +00:00
Glyn Normington
3a43714908 Rename Python code
The name dark_visitors.py gives the impression that the code is entirely
related to the dark visitors website, whereas the update command relates
to dark visitors and the convert command is unrelated to dark visitors.
2025-01-04 04:55:34 +00:00
fabianegli
3ab22bc498 make conversions and updates separately triggerable 2024-10-19 19:56:41 +02:00
fabianegli
6ab8fb2d37 no more failure when run without network 2024-10-19 19:11:01 +02:00
fabianegli
0c05461f84 simplify repo and added some tests 2024-10-19 13:06:34 +02:00
Glyn Normington
38a388097c Fix typo and trigger rerun of main job 2024-10-19 04:42:27 +01:00
Glyn Normington
bdf30be7dc Dump out file contents in PHP script 2024-10-19 04:33:46 +01:00
Ivan Sagalaev
7bb5efd462
Sort the content case-insensitively before dumping to JSON 2024-10-17 21:08:43 -04:00
407b9e12e6
chore: sort output 2024-08-14 17:10:29 -07:00
Chenghao Mou
944bee0f56 call main after update 2024-08-07 11:31:58 +01:00
Chenghao Mou
2a3685385c restrict scope 2024-08-06 19:33:49 +01:00
Chenghao Mou
55e92f4324 update existing ones 2024-08-06 17:48:06 +01:00
Chenghao Mou
8ab1e30a6c test workflow 2024-08-06 17:12:26 +01:00
Chenghao Mou
192bf67631 add dark visitor workflow 2024-08-06 17:02:23 +01:00
Adam Newbold
1fdc79dacb Adding GitHub Action 2024-08-01 18:17:19 -04:00