ai.robots.txt

cdransf/ai.robots.txt

Fork 0

mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-19 00:33:11 +00:00

Commit graph

d05530f116

Merge 5fba0b746d into 7a2e6cba52 Cory Dransfeldt 2025-05-18 18:58:47 +02:00
24c19117a2

Merge fd41de8522 into 7a2e6cba52 Kyle Buckingham 2025-05-18 12:19:20 +02:00
08909d958a

fix(robots.json): remove duplicate entry for Meta-ExternalFetcher meta-externalagent Cory Dransfeldt 2025-05-17 13:34:58 -07:00
330d5c17bd

Merge 6c3ae6eb20 into 7a2e6cba52 Falko Zurell 2025-05-17 13:25:57 -07:00
7a2e6cba52 Update from Dark Visitors main dark-visitors 2025-05-17 00:57:28 +00:00
6c3ae6eb20

Merge branch 'ai-robots-txt:main' into main Falko Zurell 2025-05-16 13:52:02 +02:00
684d11d889 moved traefik manual setup into docs Falko Zurell 2025-05-16 13:50:58 +02:00
dd1ed174b7 Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ai.robots.txt 2025-05-16 11:35:15 +00:00
89c0fbaf86

Merge pull request #129 from ai-robots-txt/google-cloudvertexbot Glyn Normington 2025-05-16 12:35:04 +01:00
9b5f75e2f3

Create traefik-manual-setup.md Falko Zurell 2025-05-16 13:27:31 +02:00
ca918a963f

chore(robots.json): adds Google-CloudVertexBot Cory Dransfeldt 2025-05-15 21:16:49 -07:00
8a44daa68a

fix(robots.json): remove duplicate entry for Meta-ExternalAgent Cory Dransfeldt 2025-05-15 21:00:23 -07:00
5fba0b746d

chore(robots.json): adds MistralAI-User/1.0 crawler mistral-bot Cory Dransfeldt 2025-05-15 15:40:46 -07:00
16d1de7094 Update from Dark Visitors dark-visitors 2025-05-16 00:59:08 +00:00
73f6f67adf

Merge pull request #125 from holysoles/lint_robots_json v1.30 Glyn Normington 2025-05-15 17:26:15 +01:00
498aa50760 lint robots.json during pull requests Patrick Evans 2025-05-15 11:10:06 -05:00
1c470babbe Merge pull request #123 from joehoyle/patch-1 ai.robots.txt 2025-05-15 16:12:30 +00:00
84d63916d2

Merge pull request #123 from joehoyle/patch-1 Adam Newbold 2025-05-15 12:12:21 -04:00
0c56b96fd9

Fix JSON syntax error Joe Hoyle 2025-05-15 11:26:47 -04:00
28e69e631b

Merge pull request #122 from ai-robots-txt/qualified-bot Cory Dransfeldt 2025-05-15 07:17:51 -07:00
9539256cb3

chore(robots.json): adds QualifiedBot crawler Cory Dransfeldt 2025-05-14 16:46:32 -07:00
9659c88b0c

Merge pull request #121 from solution-libre/add-traefik-plugin Cory Dransfeldt 2025-05-14 16:45:34 -07:00
c66d180295

Merge branch 'main' into add-traefik-plugin Florent Poinsaut 2025-05-14 22:06:56 +02:00
9a9b1b41c0

Merge pull request #119 from ai-robots-txt/bing-ai-opt-out-instructions Glyn Normington 2025-05-14 19:18:20 +01:00
b4610a725c Add Traefik plugin Florent Poinsaut 2025-05-14 14:11:56 +02:00
36a52a88d8

Bing AI opt-out instructions Cory Dransfeldt 2025-05-12 20:20:18 -07:00
5fe40a1397

Update robots.txt (Added GoogleBot and PiplBot) MSK 2025-05-12 18:45:54 +06:00
76f8be7d03 Delete unreferenced variable Glyn Normington 2025-05-01 11:30:11 +01:00
678380727e Merge pull request #115 from glyn/syntax ai.robots.txt 2025-05-01 10:29:06 +00:00
fb8188c49d

Merge pull request #115 from glyn/syntax Glyn Normington 2025-05-01 11:28:54 +01:00
ec995cd686 Fix Python syntax error Glyn Normington 2025-05-01 11:27:40 +01:00
1310dbae46

Added a Caddyfile converter (#110) Crazyroostereye 2025-05-01 12:21:32 +02:00
0c697d7c6a

Fix typo Glyn Normington 2025-05-01 11:20:09 +01:00
6baa7725b3

Merge branch 'main' into main Crazyroostereye 2025-04-30 22:13:51 +02:00
8164cec0c8

Update bingbot-extrastep.md Dennis Lee 2025-04-29 17:56:14 +01:00
bd5fb8a37f

Bing AI opt-out instructions Dennis Lee 2025-04-28 22:02:12 +01:00
91a88e2fa8

Merge pull request #113 from rwijnen-um/feature/haproxy Glyn Normington 2025-04-28 09:00:16 +01:00
a4a9f2ac2b Tests for HAProxy file added. Rik Wijnen 2025-04-28 09:30:26 +02:00
66da70905f Fixed incorrect English sentence. Rik Wijnen 2025-04-28 09:09:40 +02:00
50e739dd73 HAProxy converter added. Rik Wijnen 2025-04-28 08:42:52 +02:00
df6d7f9723 Accedently change to the wrong Commit Julian Beittel 2025-04-27 12:19:30 +02:00
daa0d80c95 Fixed Diff Julian Beittel 2025-04-27 12:16:14 +02:00
c6c7f1748f Update from Dark Visitors ai.robots.txt 2025-04-26 00:55:12 +00:00
934ac7b318 Update from Dark Visitors dark-visitors 2025-04-25 00:56:57 +00:00
4654e14e9c Merge pull request #112 from maiavixen/main ai.robots.txt 2025-04-24 07:00:34 +00:00
9bf31fbca8

Merge pull request #112 from maiavixen/main Glyn Normington 2025-04-24 08:00:24 +01:00
9d846ced45

Update robots.json maia 2025-04-24 04:08:20 +02:00
1e92364c13 Added Test and Docu Julian Beittel 2025-04-23 09:41:18 +02:00
5864d9db4e

Merge d79ca19f38 into 8d25a424d9 Katrin Leinweber 2025-04-23 09:28:07 +02:00
8d25a424d9 Update from Dark Visitors dark-visitors 2025-04-23 00:56:52 +00:00
e519f9f94c Added a Caddyfile converter Julian Beittel 2025-04-22 20:17:37 +02:00
bbec639c14 Merge pull request #109 from dennislee1/patch-1 v1.29 ai.robots.txt 2025-04-22 14:50:26 +00:00
422cf9e29b

Merge pull request #109 from dennislee1/patch-1 Cory Dransfeldt 2025-04-22 07:50:14 -07:00
33c5ce1326

Update robots.json Dennis Lee 2025-04-21 18:55:11 +01:00
774b1ddf52

Merge pull request #107 from glyn/sponsorship Cory Dransfeldt 2025-04-18 11:40:06 -07:00
b1856e6988 Donations Glyn Normington 2025-04-18 18:40:44 +01:00
d05ede8fe1 Clarify our position on sponsorship Glyn Normington 2025-04-18 17:46:56 +01:00
fd41de8522

Update robots.json Kyle Buckingham 2025-04-16 16:43:03 -07:00
4a6f37d727

Update robots.json Kyle Buckingham 2025-04-16 16:42:58 -07:00
e0cdb278fb Update from Dark Visitors v1.28 ai.robots.txt 2025-04-16 00:57:11 +00:00
a96e330989 Update from Dark Visitors dark-visitors 2025-04-15 00:57:01 +00:00
156e6baa09

Merge pull request #105 from jsheard/patch-1 Cory Dransfeldt 2025-04-14 10:08:38 -07:00
d9f882a9b2

Include "AI Agents" from Dark Visitors Joshua Sheard 2025-04-14 15:46:01 +01:00
305188b2e7 Update from Dark Visitors dark-visitors 2025-04-11 00:55:52 +00:00
4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot v1.27 ai.robots.txt 2025-04-10 19:22:34 +00:00
a891ad7213

Merge pull request #102 from ai-robots-txt/imgproxy-bot Cory Dransfeldt 2025-04-10 12:22:23 -07:00
b65f45e408

chore(robots.json): adds imgproxy crawler Cory Dransfeldt 2025-04-10 10:12:51 -07:00
49e58b1573

Merge pull request #100 from fbartho/fb/fix-perplexity-users Glyn Normington 2025-04-05 17:32:19 +01:00
c6f308cbd0

PR Feedback: log special-case, comment consistency Frederic Barthelemy 2025-04-05 09:01:52 -07:00
5f5a89c38c

Fix html-mangled hyphen in Perplexity-Users Frederic Barthelemy 2025-04-04 17:34:14 -07:00
6b0349f37d

fix python complaining about f-string syntax Frederic Barthelemy 2025-04-04 15:20:30 -07:00
8dc36aa2e2

Update robots.txt Kyle Buckingham 2025-04-01 15:23:28 -07:00
ae8f74c10c

Update robots.json Kyle Buckingham 2025-04-01 15:22:04 -07:00
b3d057f93f

Update claude bots Kyle Buckingham 2025-04-01 15:13:51 -07:00
5b8650b99b Update from Dark Visitors ai.robots.txt 2025-03-29 00:54:10 +00:00
c249de99a3 Update from Dark Visitors dark-visitors 2025-03-28 00:54:28 +00:00
ec18af7624

Revert "Merge pull request #91 from deyigifts/perplexity-user" Cory Dransfeldt 2025-03-27 12:51:22 -07:00
6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export ai.robots.txt 2025-03-27 19:49:15 +00:00
dba03d809c

Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export Glyn Normington 2025-03-27 19:49:05 +00:00
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user ai.robots.txt 2025-03-27 19:29:30 +00:00
1183187be9

Merge pull request #91 from deyigifts/perplexity-user Cory Dransfeldt 2025-03-27 12:29:21 -07:00
7c3b5a2cb2

Add tests for Nginx config generator Thomas Leister 2025-03-27 16:12:18 +01:00
d79ca19f38 Add Lightpanda due to its AI/LLM focus Katrin Leinweber 2025-03-27 17:59:09 +01:00
4f3f4cd0dd

Add assembled version of nginx-block-ai-bots.conf file Thomas Leister 2025-03-27 12:28:50 +01:00
5a312c5f4d

Mention Nginx config feature in README Thomas Leister 2025-03-27 12:28:11 +01:00
da85207314

Implement new function "json_to_nginx" which outputs an Nginx configuration snippet Thomas Leister 2025-03-27 12:27:09 +01:00
84f4ed85a6

Update robots.json katagaki 2025-03-26 13:59:27 +09:00
35a461ff18

Update robots.json katagaki 2025-03-26 13:58:58 +09:00
9f506ac0aa

Add Kagibot to robots.txt katagaki 2025-03-26 13:52:10 +09:00
6ecfcdfcbf

Update perplexity bot deyigifts 2025-03-24 14:16:57 +08:00
e5bf972187

Added AhrefsBot theRealProHacker 2025-02-28 13:40:37 +01:00
3be8edc5e4

Added bing bots (see https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) theRealProHacker 2025-02-28 13:31:02 +01:00
33288d3e23

Added SemrushBot/7~bl theRealProHacker 2025-02-28 13:20:19 +01:00
5e7c3c432f

Merge pull request #83 from glyn/81-doc-testing Cory Dransfeldt 2025-02-19 09:19:44 -08:00
9f41d4c11c

Merge pull request #84 from sideeffect42/tests-workflow Glyn Normington 2025-02-18 19:42:55 +00:00
8a74896333 Add workflow to run tests on pull request or push to main Dennis Camera 2025-02-18 10:23:40 +01:00
1d55a205e4 Document testing in README Glyn Normington 2025-02-18 05:08:28 +00:00
8494a7fcaa

Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt Glyn Normington 2025-02-18 16:42:36 +00:00
c7c1e7b96f robots.py: Make executable Dennis Camera 2025-02-18 10:15:10 +01:00
17b826a6d3 Update tests and convert to stock unittest Dennis Camera 2025-02-18 10:13:27 +01:00