ai.robots.txt
6d9ce1d62a
chore: add sidetrade bot
2024-09-28 20:58:18 +00:00
6a988be27f
chore: add sidetrade bot
2024-09-28 13:58:00 -07:00
ai.robots.txt
632e9d6510
Daily update from Dark Visitors
2024-09-28 01:17:19 +00:00
dark-visitors
7851cea4fd
Daily update from Dark Visitors
2024-09-27 01:18:04 +00:00
Glyn Normington
75343c790e
Merge pull request #38 from urvish-p80/main
...
Add an additional resource - README.md
2024-09-27 01:26:04 +01:00
ai.robots.txt
44d975c799
Merge pull request #42 from commoncrawl/main
...
feat: make CCBot entry more accurate
2024-09-27 00:21:49 +00:00
Glyn Normington
2f67e77ddb
Merge pull request #42 from commoncrawl/main
...
feat: make CCBot entry more accurate
2024-09-27 01:21:37 +01:00
Greg Lindahl
a6de89e6bd
feat: make CCBot entry more accurate
2024-09-26 21:41:28 +00:00
60bdfa7eb3
Merge pull request #41 from cityrolr/patch-1
...
Update README.md
2024-09-24 12:53:52 -07:00
Julian Mair
af05890b07
Update README.md
...
For people who don't use or don't want to use RSS for this, I've added a little explanation of how to subscribe to releases via GitHub.
2024-09-23 23:27:27 +02:00
Urvish Patel
0106d4b15a
Add additional resource - README.md
...
A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt.
2024-09-23 08:19:27 -04:00
ai.robots.txt
6b8d7f5890
Daily update from Dark Visitors
2024-09-09 01:16:21 +00:00
dark-visitors
5963cbf9f7
Daily update from Dark Visitors
2024-09-08 01:19:31 +00:00
Glyn Normington
b15b8062ce
Merge pull request #36 from cramforce/patch-1
...
Add instructions for AI bot blocking on Vercel
2024-09-08 01:26:07 +01:00
Malte Ubl
809851ae88
Add instructions for AI bot blocking on Vercel
2024-09-07 15:59:25 -07:00
ai.robots.txt
1c1b423684
chore: add iaskspider/2.0
2024-09-07 02:05:43 +00:00
8373294404
chore: add iaskspider/2.0
2024-09-06 19:05:26 -07:00
b30ca5f193
Merge pull request #35 from nisbet-hubbard/patch-7
...
Improve main workflow
2024-09-02 18:40:57 -07:00
ai.robots.txt
fb5c995243
Daily update from Dark Visitors
2024-09-03 01:12:57 +00:00
ai.robots.txt
7151f6c569
Removing previously generated files
2024-09-03 01:12:56 +00:00
nisbet-hubbard
cc18b8617c
Update main.yml
2024-09-03 07:48:48 +08:00
ai.robots.txt
c9325c9e18
Daily update from Dark Visitors
2024-09-02 01:15:07 +00:00
ai.robots.txt
567bd00aec
Removing previously generated files
2024-09-02 01:15:07 +00:00
ai.robots.txt
543e993b08
Daily update from Dark Visitors
2024-09-01 01:24:53 +00:00
ai.robots.txt
01589718df
Removing previously generated files
2024-09-01 01:24:52 +00:00
ai.robots.txt
9a7f556d87
Daily update from Dark Visitors
2024-08-31 01:13:04 +00:00
ai.robots.txt
9a4ebb57ee
Removing previously generated files
2024-08-31 01:13:04 +00:00
ai.robots.txt
054c97ad4f
Daily update from Dark Visitors
2024-08-30 01:13:29 +00:00
ai.robots.txt
b2970316d8
Removing previously generated files
2024-08-30 01:13:29 +00:00
ai.robots.txt
008a34ceb4
chore: add ai2bot
2024-08-29 03:07:52 +00:00
ai.robots.txt
3bce634e4a
Removing previously generated files
2024-08-29 03:07:51 +00:00
0f8723558f
chore: add ai2bot
2024-08-28 20:07:32 -07:00
ai.robots.txt
6dc900b582
Daily update from Dark Visitors
2024-08-29 01:13:19 +00:00
ai.robots.txt
71eefcdb05
Removing previously generated files
2024-08-29 01:13:19 +00:00
ai.robots.txt
1d417ffab9
Daily update from Dark Visitors
2024-08-28 01:12:35 +00:00
ai.robots.txt
00ef18f93c
Removing previously generated files
2024-08-28 01:12:35 +00:00
ai.robots.txt
84a2376f65
Daily update from Dark Visitors
2024-08-27 01:12:20 +00:00
ai.robots.txt
699862f4bd
Removing previously generated files
2024-08-27 01:12:19 +00:00
ai.robots.txt
ccec3eef15
Daily update from Dark Visitors
2024-08-26 01:11:41 +00:00
ai.robots.txt
6cb9bc8ebf
Removing previously generated files
2024-08-26 01:11:40 +00:00
ai.robots.txt
42a7ca7eda
Daily update from Dark Visitors
2024-08-25 01:16:28 +00:00
ai.robots.txt
907866301f
Removing previously generated files
2024-08-25 01:16:27 +00:00
ai.robots.txt
b202b9e1e3
Daily update from Dark Visitors
2024-08-24 01:09:29 +00:00
ai.robots.txt
ac1250cfa5
Removing previously generated files
2024-08-24 01:09:29 +00:00
ai.robots.txt
d95f2e8072
Daily update from Dark Visitors
2024-08-23 01:10:54 +00:00
ai.robots.txt
61d851baf5
Removing previously generated files
2024-08-23 01:10:53 +00:00
dark-visitors
7bfc1647a8
Daily update from Dark Visitors
2024-08-22 01:11:43 +00:00
ai.robots.txt
3580a7096f
Daily update from Dark Visitors
2024-08-21 01:10:11 +00:00
ai.robots.txt
fad335178f
Removing previously generated files
2024-08-21 01:10:10 +00:00
ai.robots.txt
358df0833e
Daily update from Dark Visitors
2024-08-20 01:10:11 +00:00