Chenghao Mou
|
192bf67631
|
add dark visitor workflow
|
2024-08-06 17:02:23 +01:00 |
|
ai.robots.txt
|
e12ddc0f42
|
Merge pull request #29 from jbowdre/dev
only build on changes to robots.json
|
2024-08-06 15:44:54 +00:00 |
|
ai.robots.txt
|
b54e274bbc
|
Removing previously generated files
|
2024-08-06 15:44:53 +00:00 |
|
|
|
3e91a84d11
|
Merge pull request #29 from jbowdre/dev
only build on changes to robots.json
|
2024-08-04 16:04:59 -07:00 |
|
John Bowdre
|
b0a93aeb70
|
only build on changes to robots.json
|
2024-08-04 17:45:18 -05:00 |
|
ai.robots.txt
|
eb924b9856
|
Merge pull request #28 from jsheard/patch-2
Add Cloudflares first-party scraper blocking to FAQ
|
2024-08-04 21:54:17 +00:00 |
|
ai.robots.txt
|
1cfc071498
|
Removing previously generated files
|
2024-08-04 21:54:16 +00:00 |
|
|
|
24c3509a6e
|
Merge pull request #28 from jsheard/patch-2
Add Cloudflares first-party scraper blocking to FAQ
|
2024-08-04 14:54:06 -07:00 |
|
ai.robots.txt
|
c2f177870f
|
Merge pull request #27 from jsheard/patch-1
Fix Imagesift user agent
|
2024-08-04 21:53:48 +00:00 |
|
ai.robots.txt
|
0072b8f5f0
|
Removing previously generated files
|
2024-08-04 21:53:47 +00:00 |
|
|
|
9c7257e7cf
|
Merge pull request #27 from jsheard/patch-1
Fix Imagesift user agent
|
2024-08-04 14:53:36 -07:00 |
|
Joshua Sheard
|
8dbbdbf44c
|
Add Cloudflares first-party scraper blocking to FAQ
|
2024-08-04 21:38:02 +01:00 |
|
Joshua Sheard
|
146fd4ffba
|
Fix Imagesift user agent
|
2024-08-04 21:33:04 +01:00 |
|
ai.robots.txt
|
c7b781034e
|
chore: restore FriendlyCrawler + ImageSift
|
2024-08-04 19:29:01 +00:00 |
|
ai.robots.txt
|
9a8fa66772
|
Removing previously generated files
|
2024-08-04 19:29:00 +00:00 |
|
|
|
1ca936ce11
|
chore: restore FriendlyCrawler + ImageSift
|
2024-08-04 12:28:48 -07:00 |
|
ai.robots.txt
|
8de5bc8e01
|
Merge pull request #25 from mirium999/add_icc_crawler
Add ICC-Crawler
|
2024-08-04 01:21:56 +00:00 |
|
ai.robots.txt
|
8c632e1ba4
|
Removing previously generated files
|
2024-08-04 01:21:55 +00:00 |
|
Adam Newbold
|
8d4d52cdab
|
Merge pull request #25 from mirium999/add_icc_crawler
Add ICC-Crawler
|
2024-08-03 21:21:45 -04:00 |
|
Mirium999
|
5826c18909
|
Add ICC-Crawler
|
2024-08-04 10:11:25 +09:00 |
|
ai.robots.txt
|
ffbad453f3
|
Merge pull request #24 from nisbet-hubbard/patch-5
Add last line of defence to FAQ
|
2024-08-03 14:27:47 +00:00 |
|
ai.robots.txt
|
b1907d86be
|
Removing previously generated files
|
2024-08-03 14:27:46 +00:00 |
|
|
|
55c585e9e3
|
Merge pull request #24 from nisbet-hubbard/patch-5
Add last line of defence to FAQ
|
2024-08-03 07:27:37 -07:00 |
|
nisbet-hubbard
|
2b56c72bac
|
Update FAQ.md
|
2024-08-03 14:27:25 +08:00 |
|
nisbet-hubbard
|
b24e5cb3bb
|
Update FAQ.md
|
2024-08-03 14:12:50 +08:00 |
|
nisbet-hubbard
|
74b1502839
|
Update FAQ.md
|
2024-08-03 14:04:58 +08:00 |
|
ai.robots.txt
|
d8de1ebdd5
|
chore: contribution note
|
2024-08-02 16:32:00 +00:00 |
|
ai.robots.txt
|
9d8d3de8ed
|
Removing previously generated files
|
2024-08-02 16:31:59 +00:00 |
|
|
|
349c35eed6
|
chore: contribution note
|
2024-08-02 09:31:48 -07:00 |
|
ai.robots.txt
|
b144225ece
|
chore: drop in additional data
|
2024-08-01 22:33:23 +00:00 |
|
ai.robots.txt
|
06b950bce9
|
Removing previously generated files
|
2024-08-01 22:33:23 +00:00 |
|
|
|
b20dfec1e4
|
chore: drop in additional data
|
2024-08-01 15:33:07 -07:00 |
|
ai.robots.txt
|
f18f0d99b9
|
chore: remove test data
|
2024-08-01 22:29:02 +00:00 |
|
ai.robots.txt
|
747cc834c4
|
Removing previously generated files
|
2024-08-01 22:29:01 +00:00 |
|
|
|
efabf3e721
|
chore: remove test data
|
2024-08-01 15:25:55 -07:00 |
|
Adam Newbold
|
1fdc79dacb
|
Adding GitHub Action
|
2024-08-01 18:17:19 -04:00 |
|
|
|
17a84f2c2d
|
chore: update robots table
|
2024-08-01 15:06:49 -07:00 |
|
|
|
6c596a50ea
|
chore: move FAQ into repo
|
2024-08-01 07:53:43 -07:00 |
|
|
|
6a8e7a8eb0
|
Merge pull request #22 from nisbet-hubbard/patch-4
Add `PetalBot` (and `facebookexternalhit`?)
|
2024-08-01 07:49:30 -07:00 |
|
nisbet-hubbard
|
df89722038
|
Add PetalBot (and facebookexternalhit?)
|
2024-07-31 18:27:29 +08:00 |
|
|
|
fa7b64ae4b
|
chore: add Scrapy
|
2024-07-30 10:28:46 -07:00 |
|
|
|
55b4505e30
|
chore: add Timpibot
|
2024-07-29 12:38:22 -07:00 |
|
|
|
d49e860b74
|
chore: add VelenPublicWebCrawler
|
2024-07-29 12:12:42 -07:00 |
|
|
|
6e323554c6
|
chore: add Meta-ExternalAgent
|
2024-07-29 08:27:31 -07:00 |
|
|
|
2972926532
|
chore: add OAI-SearchBot
|
2024-07-26 09:06:10 -07:00 |
|
Glyn Normington
|
c17cae6e9d
|
link to bot metrics table
Make it easier to view the table.
|
2024-07-17 02:28:32 +01:00 |
|
|
|
3692d66918
|
chore: update bots table
|
2024-07-16 12:07:01 -07:00 |
|
|
|
af52578965
|
chore: drop google adbot; add GoogleOther bots
|
2024-07-16 12:05:34 -07:00 |
|
|
|
570fd36ea2
|
chore: update bots table
|
2024-07-10 19:47:23 -07:00 |
|
|
|
0ca6bce87e
|
chore: add ImagesiftBot
|
2024-07-09 17:41:32 -07:00 |
|
