Dennis Camera
0bd3fa63b8
table-of-bot-metrics.md: Escape robot names for Markdown table
...
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.
The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9
.htaccess: Make regex in RewriteCond safe
...
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87
.htaccess: Allow robots access to /robots.txt
2025-02-18 12:49:29 +01:00
dark-visitors
abfd6dfcd1
Update from Dark Visitors
2025-02-17 00:53:32 +00:00
ai.robots.txt
693289bb29
chore: add Brightbot 1.0
2025-02-16 21:37:52 +00:00
a9ec4ffa6f
chore: add Brightbot 1.0
2025-02-16 13:36:39 -08:00
Glyn Normington
03aa829913
Merge pull request #79 from always-be-testing/main
...
List of AI bots Cloudflare considers "Verified"
2025-02-16 04:33:40 +00:00
always-be-testing
5b13c2e504
add more concise message about verified bots
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-02-15 11:22:10 -05:00
always-be-testing
af87b85d7f
include return after heading
2025-02-14 12:39:08 -05:00
always-be-testing
f99339922f
grammar update and include syntax for verified bot condition
2025-02-14 12:36:33 -05:00
always-be-testing
e396a2ec78
forgot to include heading
2025-02-14 12:31:20 -05:00
always-be-testing
261a2b83b9
update README to inclide list of ai bots Cloudflare considers verified
2025-02-14 12:26:19 -05:00
dark-visitors
bebffccc0c
Update from Dark Visitors
2025-02-02 00:52:50 +00:00
ai.robots.txt
89d4c6e5ca
Merge pull request #73 from nisbet-hubbard/patch-8
...
Actually block Semrush’s AI tools
2025-02-01 10:51:01 +00:00
Glyn Normington
f9e2c5810b
Merge pull request #73 from nisbet-hubbard/patch-8
...
Actually block Semrush’s AI tools
2025-02-01 10:50:50 +00:00
nisbet-hubbard
05b79b8a58
Update robots.json
2025-01-27 19:41:03 +08:00
dark-visitors
9c060dee1c
Update from Dark Visitors
2025-01-21 00:49:22 +00:00
ai.robots.txt
6c552a3daa
Merge pull request #71 from jsheard/patch-1
...
Add Crawlspace
2025-01-20 17:45:42 +00:00
Glyn Normington
f621fb4852
Merge pull request #71 from jsheard/patch-1
...
Add Crawlspace
2025-01-20 17:45:29 +00:00
Joshua Sheard
7427d96bac
Update robots.json
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 10:59:02 +00:00
Glyn Normington
81cc81b35e
Merge pull request #68 from MassiminoilTrace/main
...
Implementing automatic htaccess generation
2025-01-20 07:33:54 +00:00
Massimo Gismondi
4f03818280
Removed if condition and added a little comments
2025-01-20 06:51:06 +01:00
Massimo Gismondi
a9956f7825
Removed additional sections
2025-01-20 06:50:48 +01:00
Massimo Gismondi
33c38ee70b
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:28:32 +01:00
Massimo Gismondi
52241bdca6
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:27:56 +01:00
Massimo Gismondi
013b7abfa1
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:27:02 +01:00
Massimo Gismondi
70fd6c0fb1
Add mention of htaccess in readme
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:25:07 +01:00
Joshua Sheard
5aa08bc002
Add Crawlspace
2025-01-19 22:03:50 +00:00
Massimo Gismondi
d65128d10a
Removed paragraph in favour of future FAQ.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:41:09 +01:00
Massimo Gismondi
1cc4b59dfc
Shortened htaccess instructions
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:40:03 +01:00
Massimo Gismondi
8aee2f24bb
Fixed space in comment
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:39:07 +01:00
Massimo Gismondi
b455af66e7
Adding clarification about performance and code comment
2025-01-17 21:42:08 +01:00
Massimo Gismondi
189e75bbfd
Adding usage instructions
2025-01-17 21:25:23 +01:00
Massimo Gismondi
933aa6159d
Implementing htaccess generation
2025-01-07 11:02:29 +01:00
Glyn Normington
b7f908e305
Merge pull request #66 from fabianegli/patch-1
...
Allow Action to succeed even if no changes were made
2025-01-07 03:54:40 +00:00
ai.robots.txt
ec454b71d3
Merge pull request #67 from Nightfirecat/semrushbot
...
Block SemrushBot
2025-01-06 20:51:56 +00:00
565dca3dc0
Merge pull request #67 from Nightfirecat/semrushbot
...
Block SemrushBot
2025-01-06 12:51:43 -08:00
Jordan Atwood
143f8f2285
Block SemrushBot
2025-01-06 12:34:38 -08:00
8e98cc6049
Merge pull request #61 from glyn/improve-naming
...
Rename Python code
2025-01-06 08:10:47 -08:00
Fabian Egli
30ee957011
bail when NO changes are staged
2025-01-06 12:05:42 +01:00
Fabian Egli
83cd546470
allow Action to succeed even if no changes were made
...
Before, the Action would fail in case there were no changes made to any files by the converter.
2025-01-06 11:39:41 +01:00
ai.robots.txt
ca8620e28b
Merge pull request #63 from glyn/push-paths
...
Convert robots.json more frequently
2025-01-05 05:05:20 +00:00
Glyn Normington
b9df958b39
Merge pull request #63 from glyn/push-paths
...
Convert robots.json more frequently
2025-01-05 05:05:01 +00:00
Glyn Normington
c01a684036
Convert robots.json more frequently
...
Specifically, when github workflows or code
is changed as either of these can affect the
conversion results.
Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/60
2025-01-05 05:03:50 +00:00
Glyn Normington
d2be15447c
Merge pull request #62 from ai-robots-txt/missing-dependency
...
Ensure dependency installed
2025-01-05 01:46:27 +00:00
Glyn Normington
9e372d0696
Ensure dependency installed
...
Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/60#issuecomment-2571437913
Ref: https://stackoverflow.com/questions/11783875/importerror-no-module-named-bs4-beautifulsoup
2025-01-05 01:45:33 +00:00
Glyn Normington
996b9c678c
Improve job name
...
The purpose of the job is to convert the JSON file
to the other files.
2025-01-04 05:28:41 +00:00
Glyn Normington
e4c12ee2f8
Rename in test code
2025-01-04 05:03:48 +00:00
Glyn Normington
3a43714908
Rename Python code
...
The name dark_visitors.py gives the impression that the code is entirely
related to the dark visitors website, whereas the update command relates
to dark visitors and the convert command is unrelated to dark visitors.
2025-01-04 04:55:34 +00:00
dark-visitors
2036a68c1f
Update from Dark Visitors
2024-12-04 00:55:50 +00:00
