ai.robots.txt
|
d8de1ebdd5
|
chore: contribution note
|
2024-08-02 16:32:00 +00:00 |
|
ai.robots.txt
|
9d8d3de8ed
|
Removing previously generated files
|
2024-08-02 16:31:59 +00:00 |
|
ai.robots.txt
|
b144225ece
|
chore: drop in additional data
|
2024-08-01 22:33:23 +00:00 |
|
ai.robots.txt
|
06b950bce9
|
Removing previously generated files
|
2024-08-01 22:33:23 +00:00 |
|
ai.robots.txt
|
f18f0d99b9
|
chore: remove test data
|
2024-08-01 22:29:02 +00:00 |
|
ai.robots.txt
|
747cc834c4
|
Removing previously generated files
|
2024-08-01 22:29:01 +00:00 |
|
|
|
17a84f2c2d
|
chore: update robots table
|
2024-08-01 15:06:49 -07:00 |
|
|
|
fa7b64ae4b
|
chore: add Scrapy
|
2024-07-30 10:28:46 -07:00 |
|
|
|
55b4505e30
|
chore: add Timpibot
|
2024-07-29 12:38:22 -07:00 |
|
|
|
d49e860b74
|
chore: add VelenPublicWebCrawler
|
2024-07-29 12:12:42 -07:00 |
|
|
|
6e323554c6
|
chore: add Meta-ExternalAgent
|
2024-07-29 08:27:31 -07:00 |
|
|
|
2972926532
|
chore: add OAI-SearchBot
|
2024-07-26 09:06:10 -07:00 |
|
|
|
3692d66918
|
chore: update bots table
|
2024-07-16 12:07:01 -07:00 |
|
|
|
570fd36ea2
|
chore: update bots table
|
2024-07-10 19:47:23 -07:00 |
|
Corbin Davenport
|
fe98e41546
|
Update Perplexity information
|
2024-06-21 22:17:45 -04:00 |
|
Corbin Davenport
|
149a72be0c
|
Add bot info for omgili, peer39, and youbot
|
2024-06-21 21:43:41 -04:00 |
|
Corbin Davenport
|
ec4610a118
|
Add information for Google, Meta, and img2dataset bots
|
2024-06-21 20:49:25 -04:00 |
|
|
|
a90ee5e9f0
|
chore: clean up bots and narrow scope
|
2024-06-20 08:09:21 -07:00 |
|
|
|
429336d725
|
fix: resolve conflict
|
2024-06-15 09:26:14 -07:00 |
|
|
|
3f65a93891
|
chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search
|
2024-06-15 09:25:17 -07:00 |
|
Christian Sievers
|
ce0b99124b
|
Update table-of-bot-metrics.md
add link to Apple docs and description for Applebot-Extended
|
2024-06-13 14:17:31 +02:00 |
|
|
|
1ac25b9135
|
chore: propagate applebot-extended out
|
2024-06-12 18:53:26 -07:00 |
|
|
|
7a5a21fdad
|
chore: update bots table
|
2024-05-29 09:21:31 -07:00 |
|
|
|
2ca4154d4e
|
chore: update perplexity entry
|
2024-05-29 09:14:35 -07:00 |
|
KG
|
04688e56fb
|
Update table-of-bot-metrics.md
Fix Markdown
|
2024-05-16 18:25:46 -05:00 |
|
|
|
696d2acef8
|
chore: add cohere-ai
|
2024-05-06 10:39:07 -07:00 |
|
|
|
6940db2412
|
chore: add anthropic bots to table
|
2024-05-06 10:36:16 -07:00 |
|
|
|
ec8e10a56c
|
chore: add bytespider to table
|
2024-05-06 10:34:43 -07:00 |
|
|
|
262fe71ad9
|
chore: add ccbot data to table
|
2024-05-06 10:32:05 -07:00 |
|
|
|
dea035365f
|
chore: add Diffbot and scoopit
|
2024-05-05 14:50:04 -07:00 |
|
Cory Dransfeldt
|
29e0799332
|
chore: add img2dataset to table-of-bot-metrics.md
|
2024-04-22 09:30:00 -07:00 |
|
Cory Dransfeldt
|
9be338094b
|
chore: populate gptbot
|
2024-04-10 10:53:16 -07:00 |
|
Cory Dransfeldt
|
cc5966af5f
|
chore: populate ChatGPT-User
|
2024-04-10 10:39:34 -07:00 |
|
Cory Dransfeldt
|
0a8952b34f
|
chore: add applebot
|
2024-04-10 10:18:51 -07:00 |
|
Cory Dransfeldt
|
b402ec681c
|
chore: add anthropic + amazonbot
|
2024-04-10 10:15:22 -07:00 |
|
Matthew Linares
|
aa0edf05b3
|
Add Google Gemini response on AbsBot-Google for bot metrics table.
Reference: https://g.co/gemini/share/5bef4db5f87d
|
2024-04-10 15:40:45 +01:00 |
|
Matthew Linares
|
048045ac6d
|
Adds table of bot details and metrics.
|
2024-04-08 20:29:45 +01:00 |
|
