|
|
429336d725
|
fix: resolve conflict
|
2024-06-15 09:26:14 -07:00 |
|
|
|
3f65a93891
|
chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search
|
2024-06-15 09:25:17 -07:00 |
|
|
|
ca730a81ca
|
Merge pull request #15 from christiansievers/main
Update table-of-bot-metrics.md
|
2024-06-13 10:50:10 -07:00 |
|
Christian Sievers
|
ce0b99124b
|
Update table-of-bot-metrics.md
add link to Apple docs and description for Applebot-Extended
|
2024-06-13 14:17:31 +02:00 |
|
|
|
1a4868f1a2
|
chore: update readme
|
2024-06-12 19:05:15 -07:00 |
|
|
|
b5d4d5bc74
|
chore: update readme
|
2024-06-12 19:01:23 -07:00 |
|
|
|
548a6686ab
|
chore: add rss badge
|
2024-06-12 18:56:09 -07:00 |
|
|
|
1ac25b9135
|
chore: propagate applebot-extended out
|
2024-06-12 18:53:26 -07:00 |
|
|
|
c546dc048c
|
Merge pull request #14 from chriskirknielsen/patch-1
Block Applebot-Extended
|
2024-06-12 18:51:09 -07:00 |
|
Christopher Kirk-Nielsen
|
39363fc813
|
Block Applebot-Extended
Per [Apple's docs](https://support.apple.com/en-us/119829#datausage) ([via Matthew Bogart](https://matthewbogart.com/@matt/112605297864483766))
|
2024-06-12 16:43:12 -04:00 |
|
|
|
d464f260e0
|
Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt
|
2024-06-10 13:23:55 -07:00 |
|
|
|
fd10fbda94
|
chore: add cloudflare waf expression
|
2024-06-10 13:23:47 -07:00 |
|
|
|
7a5a21fdad
|
chore: update bots table
|
2024-05-29 09:21:31 -07:00 |
|
|
|
2ca4154d4e
|
chore: update perplexity entry
|
2024-05-29 09:14:35 -07:00 |
|
|
|
e152c110c5
|
Merge pull request #10 from kg583/main
Fix Markdown in `table-of-bot-metrics.md`
|
2024-05-20 19:43:29 -07:00 |
|
KG
|
04688e56fb
|
Update table-of-bot-metrics.md
Fix Markdown
|
2024-05-16 18:25:46 -05:00 |
|
|
|
696d2acef8
|
chore: add cohere-ai
|
2024-05-06 10:39:07 -07:00 |
|
|
|
6940db2412
|
chore: add anthropic bots to table
|
2024-05-06 10:36:16 -07:00 |
|
|
|
ec8e10a56c
|
chore: add bytespider to table
|
2024-05-06 10:34:43 -07:00 |
|
|
|
262fe71ad9
|
chore: add ccbot data to table
|
2024-05-06 10:32:05 -07:00 |
|
|
|
dea035365f
|
chore: add Diffbot and scoopit
|
2024-05-05 14:50:04 -07:00 |
|
Cory Dransfeldt
|
29e0799332
|
chore: add img2dataset to table-of-bot-metrics.md
|
2024-04-22 09:30:00 -07:00 |
|
Cory Dransfeldt
|
5025939e5d
|
chore: add img2dataset to edge func
|
2024-04-22 09:28:39 -07:00 |
|
Cory Dransfeldt
|
8ddd2a0bb3
|
chore: add img2dataset to nginx.conf
|
2024-04-22 09:28:21 -07:00 |
|
Cory Dransfeldt
|
2b77b0b0ad
|
chore: add img2dataset to apache.conf
|
2024-04-22 09:27:41 -07:00 |
|
Cory Dransfeldt
|
118ec00126
|
chore: add img2dataset to robots.txt
|
2024-04-22 09:26:59 -07:00 |
|
Cory Dransfeldt
|
6f1a28e17a
|
chore: update nginx
|
2024-04-16 08:31:36 -07:00 |
|
Cory Dransfeldt
|
1664980617
|
chore: comment
|
2024-04-14 10:04:25 -07:00 |
|
Cory Dransfeldt
|
1e91744ed2
|
feat: add edge function
|
2024-04-14 10:03:40 -07:00 |
|
Cory Dransfeldt
|
9a92568266
|
Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt
|
2024-04-14 09:56:30 -07:00 |
|
Cory Dransfeldt
|
9d56001747
|
chore: move server configs
|
2024-04-14 09:56:23 -07:00 |
|
Cory Dransfeldt
|
d05a6956b8
|
Create FUNDING.yml
|
2024-04-12 11:40:44 -07:00 |
|
Cory Dransfeldt
|
0d9b75b227
|
chore: move additional resources to wiki
|
2024-04-12 09:33:19 -07:00 |
|
Cory Dransfeldt
|
9be338094b
|
chore: populate gptbot
|
2024-04-10 10:53:16 -07:00 |
|
Cory Dransfeldt
|
ef5d847e65
|
chore: add gitignore; move logo file
|
2024-04-10 10:44:07 -07:00 |
|
Cory Dransfeldt
|
cc5966af5f
|
chore: populate ChatGPT-User
|
2024-04-10 10:39:34 -07:00 |
|
Cory Dransfeldt
|
0a8952b34f
|
chore: add applebot
|
2024-04-10 10:18:51 -07:00 |
|
Cory Dransfeldt
|
b402ec681c
|
chore: add anthropic + amazonbot
|
2024-04-10 10:15:22 -07:00 |
|
Cory Dransfeldt
|
46b0532daf
|
Merge pull request #8 from mattlinares/feature/add-table-of-crawler-detail
Add Google Gemini response on AbsBot-Google for bot metrics table.
|
2024-04-10 09:08:22 -07:00 |
|
Cory Dransfeldt
|
db06035714
|
Merge branch 'main' into feature/add-table-of-crawler-detail
|
2024-04-10 09:07:45 -07:00 |
|
Matthew Linares
|
aa0edf05b3
|
Add Google Gemini response on AbsBot-Google for bot metrics table.
Reference: https://g.co/gemini/share/5bef4db5f87d
|
2024-04-10 15:40:45 +01:00 |
|
Cory Dransfeldt
|
b6be895994
|
fix: clean up syntax
|
2024-04-09 16:08:57 -07:00 |
|
Cory Dransfeldt
|
ed190d4803
|
fix: syntax
|
2024-04-09 16:08:13 -07:00 |
|
Cory Dransfeldt
|
0518b3e777
|
chore: add FriendlyCrawler to the table
|
2024-04-08 12:50:36 -07:00 |
|
Cory Dransfeldt
|
964b29d330
|
Merge pull request #6 from mattlinares/main
Adds table of bot details and metrics
|
2024-04-08 12:48:35 -07:00 |
|
Cory Dransfeldt
|
e03d5f4ec2
|
chore: add FriendlyCrawler to nginx.conf.txt
|
2024-04-08 12:41:40 -07:00 |
|
Cory Dransfeldt
|
2d4efec184
|
chore: add FriendlyCrawler to apache.conf.txt
|
2024-04-08 12:41:19 -07:00 |
|
Cory Dransfeldt
|
d6d40989f4
|
chore: add FriendlyCrawler to robots.txt
|
2024-04-08 12:40:59 -07:00 |
|
Matthew Linares
|
048045ac6d
|
Adds table of bot details and metrics.
|
2024-04-08 20:29:45 +01:00 |
|
Cory Dransfeldt
|
74e4f64d89
|
chore: create nginx.conf.txt
|
2024-04-08 12:27:59 -07:00 |
|
