|
|
1ca936ce11
|
chore: restore FriendlyCrawler + ImageSift
|
2024-08-04 12:28:48 -07:00 |
|
ai.robots.txt
|
8de5bc8e01
|
Merge pull request #25 from mirium999/add_icc_crawler
Add ICC-Crawler
|
2024-08-04 01:21:56 +00:00 |
|
ai.robots.txt
|
8c632e1ba4
|
Removing previously generated files
|
2024-08-04 01:21:55 +00:00 |
|
Adam Newbold
|
8d4d52cdab
|
Merge pull request #25 from mirium999/add_icc_crawler
Add ICC-Crawler
|
2024-08-03 21:21:45 -04:00 |
|
Mirium999
|
5826c18909
|
Add ICC-Crawler
|
2024-08-04 10:11:25 +09:00 |
|
ai.robots.txt
|
ffbad453f3
|
Merge pull request #24 from nisbet-hubbard/patch-5
Add last line of defence to FAQ
|
2024-08-03 14:27:47 +00:00 |
|
ai.robots.txt
|
b1907d86be
|
Removing previously generated files
|
2024-08-03 14:27:46 +00:00 |
|
|
|
55c585e9e3
|
Merge pull request #24 from nisbet-hubbard/patch-5
Add last line of defence to FAQ
|
2024-08-03 07:27:37 -07:00 |
|
nisbet-hubbard
|
2b56c72bac
|
Update FAQ.md
|
2024-08-03 14:27:25 +08:00 |
|
nisbet-hubbard
|
b24e5cb3bb
|
Update FAQ.md
|
2024-08-03 14:12:50 +08:00 |
|
nisbet-hubbard
|
74b1502839
|
Update FAQ.md
|
2024-08-03 14:04:58 +08:00 |
|
ai.robots.txt
|
d8de1ebdd5
|
chore: contribution note
|
2024-08-02 16:32:00 +00:00 |
|
ai.robots.txt
|
9d8d3de8ed
|
Removing previously generated files
|
2024-08-02 16:31:59 +00:00 |
|
|
|
349c35eed6
|
chore: contribution note
|
2024-08-02 09:31:48 -07:00 |
|
ai.robots.txt
|
b144225ece
|
chore: drop in additional data
|
2024-08-01 22:33:23 +00:00 |
|
ai.robots.txt
|
06b950bce9
|
Removing previously generated files
|
2024-08-01 22:33:23 +00:00 |
|
|
|
b20dfec1e4
|
chore: drop in additional data
|
2024-08-01 15:33:07 -07:00 |
|
ai.robots.txt
|
f18f0d99b9
|
chore: remove test data
|
2024-08-01 22:29:02 +00:00 |
|
ai.robots.txt
|
747cc834c4
|
Removing previously generated files
|
2024-08-01 22:29:01 +00:00 |
|
|
|
efabf3e721
|
chore: remove test data
|
2024-08-01 15:25:55 -07:00 |
|
Adam Newbold
|
1fdc79dacb
|
Adding GitHub Action
|
2024-08-01 18:17:19 -04:00 |
|
|
|
17a84f2c2d
|
chore: update robots table
|
2024-08-01 15:06:49 -07:00 |
|
|
|
6c596a50ea
|
chore: move FAQ into repo
|
2024-08-01 07:53:43 -07:00 |
|
|
|
6a8e7a8eb0
|
Merge pull request #22 from nisbet-hubbard/patch-4
Add `PetalBot` (and `facebookexternalhit`?)
|
2024-08-01 07:49:30 -07:00 |
|
nisbet-hubbard
|
df89722038
|
Add PetalBot (and facebookexternalhit?)
|
2024-07-31 18:27:29 +08:00 |
|
|
|
fa7b64ae4b
|
chore: add Scrapy
|
2024-07-30 10:28:46 -07:00 |
|
|
|
55b4505e30
|
chore: add Timpibot
|
2024-07-29 12:38:22 -07:00 |
|
|
|
d49e860b74
|
chore: add VelenPublicWebCrawler
|
2024-07-29 12:12:42 -07:00 |
|
|
|
6e323554c6
|
chore: add Meta-ExternalAgent
|
2024-07-29 08:27:31 -07:00 |
|
|
|
2972926532
|
chore: add OAI-SearchBot
|
2024-07-26 09:06:10 -07:00 |
|
Glyn Normington
|
c17cae6e9d
|
link to bot metrics table
Make it easier to view the table.
|
2024-07-17 02:28:32 +01:00 |
|
|
|
3692d66918
|
chore: update bots table
|
2024-07-16 12:07:01 -07:00 |
|
|
|
af52578965
|
chore: drop google adbot; add GoogleOther bots
|
2024-07-16 12:05:34 -07:00 |
|
|
|
570fd36ea2
|
chore: update bots table
|
2024-07-10 19:47:23 -07:00 |
|
|
|
0ca6bce87e
|
chore: add ImagesiftBot
|
2024-07-09 17:41:32 -07:00 |
|
|
|
0971af19b6
|
chore: peer39 unrelated to ai
|
2024-07-09 17:39:51 -07:00 |
|
|
|
74fa789985
|
Merge pull request #18 from glyn/add-ref
Add reference
|
2024-06-29 12:22:23 -07:00 |
|
Glyn Normington
|
4a8ce0b51d
|
Add reference
This links to https://github.com/glyn/nginx_robot_access
|
2024-06-29 12:20:38 +01:00 |
|
|
|
70e8985622
|
chore: remove unused image
|
2024-06-22 12:44:02 -07:00 |
|
|
|
fd4ade555c
|
Merge pull request #17 from corbindavenport/main
More bot information and improved README
|
2024-06-22 12:42:52 -07:00 |
|
Corbin Davenport
|
fe98e41546
|
Update Perplexity information
|
2024-06-21 22:17:45 -04:00 |
|
Corbin Davenport
|
20dc4e16ad
|
Update releases feed links
|
2024-06-21 22:02:54 -04:00 |
|
Corbin Davenport
|
149a72be0c
|
Add bot info for omgili, peer39, and youbot
|
2024-06-21 21:43:41 -04:00 |
|
Corbin Davenport
|
ec4610a118
|
Add information for Google, Meta, and img2dataset bots
|
2024-06-21 20:49:25 -04:00 |
|
|
|
4163ca92a5
|
chore: clean up unnecessary ai.txt
|
2024-06-21 08:58:44 -07:00 |
|
|
|
89de2d2d91
|
chore: resolve conflict
|
2024-06-20 08:12:24 -07:00 |
|
|
|
a980ace544
|
chore: update readme
|
2024-06-20 08:11:38 -07:00 |
|
|
|
a90ee5e9f0
|
chore: clean up bots and narrow scope
|
2024-06-20 08:09:21 -07:00 |
|
|
|
4b8499d718
|
Merge pull request #16 from nisbet-hubbard/patch-1
Remove potential false positives
|
2024-06-20 07:57:16 -07:00 |
|
nisbet-hubbard
|
56c2285462
|
Update robots.txt
|
2024-06-20 11:31:08 +08:00 |
|
