dark-visitors
9e06cf3bc9
Updated from new robots.json
2024-10-29 00:52:12 +00:00
dark-visitors
bc0a0ad0e9
Update from Dark Visitors
2024-10-29 00:52:12 +00:00
dark-visitors
fe5f407673
Update from Dark Visitors
2024-10-27 00:54:47 +00:00
Adam Newbold
a66b16827d
Merge pull request #51 from fabianegli/php-to-python-plus-tests
...
PHP to Python plus tests and stuff
2024-10-22 21:32:58 -04:00
fabianegli
3ab22bc498
make conversions and updates separately triggerable
2024-10-19 19:56:41 +02:00
fabianegli
6ab8fb2d37
no more failure when run without network
2024-10-19 19:11:01 +02:00
fabianegli
7e2b3ab037
rename action
2024-10-19 19:09:34 +02:00
fabianegli
0c05461f84
simplify repo and added some tests
2024-10-19 13:06:34 +02:00
fabianegli
6bb598820e
ignore venv
2024-10-19 11:56:00 +02:00
Glyn Normington
d62cab66c5
Merge pull request #50 from glyn/fix-typo
...
Fix typo and trigger rerun of main job
2024-10-19 04:43:09 +01:00
ai.robots.txt
6a359e7fd7
Fix typo and trigger rerun of main job
2024-10-19 03:43:00 +00:00
Glyn Normington
38a388097c
Fix typo and trigger rerun of main job
2024-10-19 04:42:27 +01:00
Glyn Normington
83c8603071
Merge pull request #49 from glyn/php-diagnostics
...
PHP diagnostics
2024-10-19 04:34:53 +01:00
ai.robots.txt
a80bd18fb8
Dump out file contents in PHP script
2024-10-19 03:34:29 +00:00
Glyn Normington
bdf30be7dc
Dump out file contents in PHP script
2024-10-19 04:33:46 +01:00
Glyn Normington
4d47b17c45
Merge pull request #47 from fabianegli/fabianegli-patch-1
...
log the diff in the update actions
2024-10-19 02:58:05 +01:00
dark-visitors
faf81efb12
Daily update from Dark Visitors
2024-10-19 01:17:15 +00:00
Fabian Egli
25adc6b802
log git repository status
2024-10-19 00:28:41 +02:00
Fabian Egli
b584f613cd
add some signposts to the log
2024-10-19 00:13:09 +02:00
Fabian Egli
b3068a8d90
add some signposts
2024-10-19 00:12:25 +02:00
Fabian Egli
a46d06d436
log changes made by the action in main.yml
2024-10-19 00:04:15 +02:00
Fabian Egli
cfaade6e2f
log the diff in the update action daily_update.yml
2024-10-19 00:01:15 +02:00
04f630f7f8
Merge pull request #45 from glyn/faq-update
...
Update the FAQ
2024-10-18 06:35:47 -07:00
Glyn Normington
898c8ab82d
Merge pull request #46 from isagalaev/case-insensitive-sorting
...
Sort the content of robots.json by keys, case-insensitively
2024-10-18 07:57:56 +01:00
Ivan Sagalaev
7bb5efd462
Sort the content case-insensitively before dumping to JSON
2024-10-17 21:08:43 -04:00
Glyn Normington
e6bb7cae9e
Augment the "why" FAQ
...
Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/40#issuecomment-2419078796
2024-10-17 12:27:05 +01:00
Glyn Normington
b229f5b936
Re-order the FAQ
...
The "why" question should come first.
2024-10-17 12:25:54 +01:00
dark-visitors
b1491d2694
Daily update from Dark Visitors
2024-10-09 01:17:37 +00:00
ai.robots.txt
9be286626d
Merge pull request #43 from lxjv/main
...
Update robots.json with Claude respect link
2024-10-08 02:30:17 +00:00
Glyn Normington
01993b98c3
Merge pull request #43 from lxjv/main
...
Update robots.json with Claude respect link
2024-10-08 03:30:07 +01:00
Laker Turner
dc15afe847
Update robots.json with Claude respect link
2024-10-07 17:38:01 +01:00
ai.robots.txt
6da804e826
chore: add ISSCyberRiskCrawler
2024-09-30 23:50:18 +00:00
9c2394f23b
chore: add ISSCyberRiskCrawler
2024-09-30 16:25:20 -07:00
ai.robots.txt
6d9ce1d62a
chore: add sidetrade bot
2024-09-28 20:58:18 +00:00
6a988be27f
chore: add sidetrade bot
2024-09-28 13:58:00 -07:00
ai.robots.txt
632e9d6510
Daily update from Dark Visitors
2024-09-28 01:17:19 +00:00
dark-visitors
7851cea4fd
Daily update from Dark Visitors
2024-09-27 01:18:04 +00:00
Glyn Normington
75343c790e
Merge pull request #38 from urvish-p80/main
...
Add an additional resource - README.md
2024-09-27 01:26:04 +01:00
ai.robots.txt
44d975c799
Merge pull request #42 from commoncrawl/main
...
feat: make CCBot entry more accurate
2024-09-27 00:21:49 +00:00
Glyn Normington
2f67e77ddb
Merge pull request #42 from commoncrawl/main
...
feat: make CCBot entry more accurate
2024-09-27 01:21:37 +01:00
Greg Lindahl
a6de89e6bd
feat: make CCBot entry more accurate
2024-09-26 21:41:28 +00:00
60bdfa7eb3
Merge pull request #41 from cityrolr/patch-1
...
Update README.md
2024-09-24 12:53:52 -07:00
Julian Mair
af05890b07
Update README.md
...
For people who don't use or don't want to use RSS for this, I've added a little explanation of how to subscribe to releases via GitHub.
2024-09-23 23:27:27 +02:00
Urvish Patel
0106d4b15a
Add additional resource - README.md
...
A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt.
2024-09-23 08:19:27 -04:00
ai.robots.txt
6b8d7f5890
Daily update from Dark Visitors
2024-09-09 01:16:21 +00:00
dark-visitors
5963cbf9f7
Daily update from Dark Visitors
2024-09-08 01:19:31 +00:00
Glyn Normington
b15b8062ce
Merge pull request #36 from cramforce/patch-1
...
Add instructions for AI bot blocking on Vercel
2024-09-08 01:26:07 +01:00
Malte Ubl
809851ae88
Add instructions for AI bot blocking on Vercel
2024-09-07 15:59:25 -07:00
ai.robots.txt
1c1b423684
chore: add iaskspider/2.0
2024-09-07 02:05:43 +00:00
8373294404
chore: add iaskspider/2.0
2024-09-06 19:05:26 -07:00
