Thomas Leister
5a312c5f4d
Mention Nginx config feature in README
2025-03-27 12:43:29 +01:00
Glyn Normington
1d55a205e4
Document testing in README
...
Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81
2025-02-18 16:49:08 +00:00
always-be-testing
5b13c2e504
add more concise message about verified bots
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-02-15 11:22:10 -05:00
always-be-testing
af87b85d7f
include return after heading
2025-02-14 12:39:08 -05:00
always-be-testing
f99339922f
grammar update and include syntax for verified bot condition
2025-02-14 12:36:33 -05:00
always-be-testing
e396a2ec78
forgot to include heading
2025-02-14 12:31:20 -05:00
always-be-testing
261a2b83b9
update README to inclide list of ai bots Cloudflare considers verified
2025-02-14 12:26:19 -05:00
Massimo Gismondi
a9956f7825
Removed additional sections
2025-01-20 06:50:48 +01:00
Massimo Gismondi
33c38ee70b
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:28:32 +01:00
Massimo Gismondi
52241bdca6
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:27:56 +01:00
Massimo Gismondi
013b7abfa1
Update README.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:27:02 +01:00
Massimo Gismondi
70fd6c0fb1
Add mention of htaccess in readme
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-20 06:25:07 +01:00
Massimo Gismondi
d65128d10a
Removed paragraph in favour of future FAQ.md
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:41:09 +01:00
Massimo Gismondi
1cc4b59dfc
Shortened htaccess instructions
...
Co-authored-by: Glyn Normington <work@underlap.org>
2025-01-18 12:40:03 +01:00
Massimo Gismondi
b455af66e7
Adding clarification about performance and code comment
2025-01-17 21:42:08 +01:00
Massimo Gismondi
189e75bbfd
Adding usage instructions
2025-01-17 21:25:23 +01:00
Glyn Normington
75343c790e
Merge pull request #38 from urvish-p80/main
...
Add an additional resource - README.md
2024-09-27 01:26:04 +01:00
Julian Mair
af05890b07
Update README.md
...
For people who don't use or don't want to use RSS for this, I've added a little explanation of how to subscribe to releases via GitHub.
2024-09-23 23:27:27 +02:00
Urvish Patel
0106d4b15a
Add additional resource - README.md
...
A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt.
2024-09-23 08:19:27 -04:00
1d3194f75d
chore: update readme
2024-08-18 11:34:43 -07:00
nisbet-hubbard
46540633ba
Update README.md
2024-08-10 08:22:28 +08:00
349c35eed6
chore: contribution note
2024-08-02 09:31:48 -07:00
Glyn Normington
c17cae6e9d
link to bot metrics table
...
Make it easier to view the table.
2024-07-17 02:28:32 +01:00
Glyn Normington
4a8ce0b51d
Add reference
...
This links to https://github.com/glyn/nginx_robot_access
2024-06-29 12:20:38 +01:00
Corbin Davenport
20dc4e16ad
Update releases feed links
2024-06-21 22:02:54 -04:00
a980ace544
chore: update readme
2024-06-20 08:11:38 -07:00
a90ee5e9f0
chore: clean up bots and narrow scope
2024-06-20 08:09:21 -07:00
1a4868f1a2
chore: update readme
2024-06-12 19:05:15 -07:00
b5d4d5bc74
chore: update readme
2024-06-12 19:01:23 -07:00
Cory Dransfeldt
0d9b75b227
chore: move additional resources to wiki
2024-04-12 09:33:19 -07:00
Cory Dransfeldt
ef5d847e65
chore: add gitignore; move logo file
2024-04-10 10:44:07 -07:00
Matthew Linares
048045ac6d
Adds table of bot details and metrics.
2024-04-08 20:29:45 +01:00
Cory Dransfeldt
d7064d23fe
chore: additional resources
2024-04-01 09:55:46 -07:00
Cory Dransfeldt
5e02ebc168
chore: update README
2024-03-29 12:14:48 -07:00
Glyn Normington
780d772d53
Scale down logo
2024-03-28 09:11:49 +00:00
Glyn Normington
a71da3ecc4
Add logo
2024-03-28 09:07:58 +00:00
Cory Dransfeldt
be9d22abed
Create README.md
2024-03-27 10:58:49 -07:00
