Commit graph

Select branches

Hide pull requests

main

missing-dependency

#1

#10

#14

#15

#16

#17

#18

#2

#21

#22

#24

#25

#27

#28

#29

#31

#32

#33

#35

#36

#37

#38

#39

#41

#42

#43

#45

#46

#47

#49

#50

#51

#54

#55

#56

#57

#58

#6

#61

#62

#63

#64

#65

#66

#67

#68

#71

#73

#79

#8

#80

#83

#84

#86

#91

#92

#94

#95

#95

#97

#98

#98

v1

v1.1

v1.10

v1.11

v1.12

v1.13

v1.14

v1.15

v1.16

v1.17

v1.18

v1.19

v1.2

v1.20

v1.21

v1.22

v1.23

v1.24

v1.25

v1.26

v1.3

v1.4

v1.5

v1.6

v1.7

v1.8

v1.9

Mono Color

• 68531dd8e2
  Merge 8dc36aa2e2 into 5b8650b99b Kyle Buckingham 2025-04-01 15:25:48 -0700
• 8dc36aa2e2
  Update robots.txt Kyle Buckingham 2025-04-01 15:23:28 -0700
• ae8f74c10c
  Update robots.json Kyle Buckingham 2025-04-01 15:22:04 -0700
• b3d057f93f
  Update claude bots Kyle Buckingham 2025-04-01 15:13:51 -0700
• f717afbefd
  Merge d79ca19f38 into 5b8650b99b Katrin Leinweber 2025-03-29 04:39:49 +0100
• 5b8650b99b Update from Dark Visitors main ai.robots.txt 2025-03-29 00:54:10 +0000
• c249de99a3 Update from Dark Visitors dark-visitors 2025-03-28 00:54:28 +0000
• ec18af7624
  Revert "Merge pull request #91 from deyigifts/perplexity-user" Cory Dransfeldt 2025-03-27 12:51:22 -0700
• 6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export ai.robots.txt 2025-03-27 19:49:15 +0000
• dba03d809c
  Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export Glyn Normington 2025-03-27 19:49:05 +0000
• 68d1d93714 Merge pull request #91 from deyigifts/perplexity-user ai.robots.txt 2025-03-27 19:29:30 +0000
• 1183187be9
  Merge pull request #91 from deyigifts/perplexity-user Cory Dransfeldt 2025-03-27 12:29:21 -0700
• 7c3b5a2cb2
  Add tests for Nginx config generator Thomas Leister 2025-03-27 16:12:18 +0100
• d79ca19f38 Add Lightpanda due to its AI/LLM focus Katrin Leinweber 2025-03-27 17:59:09 +0100
• 4f3f4cd0dd
  Add assembled version of nginx-block-ai-bots.conf file Thomas Leister 2025-03-27 12:28:50 +0100
• 5a312c5f4d
  Mention Nginx config feature in README Thomas Leister 2025-03-27 12:28:11 +0100
• da85207314
  Implement new function "json_to_nginx" which outputs an Nginx configuration snippet Thomas Leister 2025-03-27 12:27:09 +0100
• 84f4ed85a6
  Update robots.json katagaki 2025-03-26 13:59:27 +0900
• 35a461ff18
  Update robots.json katagaki 2025-03-26 13:58:58 +0900
• 9f506ac0aa
  Add Kagibot to robots.txt katagaki 2025-03-26 13:52:10 +0900
• 6ecfcdfcbf
  Update perplexity bot deyigifts 2025-03-24 14:16:57 +0800
• e5bf972187
  Added AhrefsBot theRealProHacker 2025-02-28 13:40:37 +0100
• 3be8edc5e4
  Added bing bots (see https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) theRealProHacker 2025-02-28 13:31:02 +0100
• 33288d3e23
  Added SemrushBot/7~bl theRealProHacker 2025-02-28 13:20:19 +0100
• 5e7c3c432f
  Merge pull request #83 from glyn/81-doc-testing Cory Dransfeldt 2025-02-19 09:19:44 -0800
• 9f41d4c11c
  Merge pull request #84 from sideeffect42/tests-workflow Glyn Normington 2025-02-18 19:42:55 +0000
• 8a74896333 Add workflow to run tests on pull request or push to main Dennis Camera 2025-02-18 10:23:40 +0100
• 1d55a205e4 Document testing in README Glyn Normington 2025-02-18 05:08:28 +0000
• 8494a7fcaa
  Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt Glyn Normington 2025-02-18 16:42:36 +0000
• c7c1e7b96f robots.py: Make executable Dennis Camera 2025-02-18 10:15:10 +0100
• 17b826a6d3 Update tests and convert to stock unittest Dennis Camera 2025-02-18 10:13:27 +0100
• 0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table Dennis Camera 2025-02-18 10:12:04 +0100
• a884a2afb9 .htaccess: Make regex in RewriteCond safe Dennis Camera 2025-02-17 21:00:57 +0100
• c0d418cd87 .htaccess: Allow robots access to /robots.txt Dennis Camera 2025-02-17 21:00:57 +0100
• abfd6dfcd1 Update from Dark Visitors dark-visitors 2025-02-17 00:53:32 +0000
• 693289bb29 chore: add Brightbot 1.0 v1.26 ai.robots.txt 2025-02-16 21:37:52 +0000
• a9ec4ffa6f
  chore: add Brightbot 1.0 Cory Dransfeldt 2025-02-16 13:36:39 -0800
• 03aa829913
  Merge pull request #79 from always-be-testing/main Glyn Normington 2025-02-16 04:33:40 +0000
• 5b13c2e504
  add more concise message about verified bots always-be-testing 2025-02-15 11:22:10 -0500
• af87b85d7f include return after heading always-be-testing 2025-02-14 12:39:08 -0500
• f99339922f grammar update and include syntax for verified bot condition always-be-testing 2025-02-14 12:36:33 -0500
• e396a2ec78 forgot to include heading always-be-testing 2025-02-14 12:31:20 -0500
• 261a2b83b9 update README to inclide list of ai bots Cloudflare considers verified always-be-testing 2025-02-14 12:26:19 -0500
• bebffccc0c Update from Dark Visitors v1.25 dark-visitors 2025-02-02 00:52:50 +0000
• 89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 ai.robots.txt 2025-02-01 10:51:01 +0000
• f9e2c5810b
  Merge pull request #73 from nisbet-hubbard/patch-8 Glyn Normington 2025-02-01 10:50:50 +0000
• 05b79b8a58
  Update robots.json nisbet-hubbard 2025-01-27 19:41:03 +0800
• 9c060dee1c Update from Dark Visitors v1.24 dark-visitors 2025-01-21 00:49:22 +0000
• 6c552a3daa Merge pull request #71 from jsheard/patch-1 ai.robots.txt 2025-01-20 17:45:42 +0000
• f621fb4852
  Merge pull request #71 from jsheard/patch-1 Glyn Normington 2025-01-20 17:45:29 +0000
• 7427d96bac
  Update robots.json Joshua Sheard 2025-01-20 10:59:02 +0000
• 81cc81b35e
  Merge pull request #68 from MassiminoilTrace/main Glyn Normington 2025-01-20 07:33:54 +0000
• 4f03818280 Removed if condition and added a little comments Massimo Gismondi 2025-01-20 06:51:06 +0100
• a9956f7825 Removed additional sections Massimo Gismondi 2025-01-20 06:50:48 +0100
• 33c38ee70b
  Update README.md Massimo Gismondi 2025-01-20 06:28:32 +0100
• 52241bdca6
  Update README.md Massimo Gismondi 2025-01-20 06:27:56 +0100
• 013b7abfa1
  Update README.md Massimo Gismondi 2025-01-20 06:27:02 +0100
• 70fd6c0fb1
  Add mention of htaccess in readme Massimo Gismondi 2025-01-20 06:25:07 +0100
• 5aa08bc002
  Add Crawlspace Joshua Sheard 2025-01-19 22:03:50 +0000
• d65128d10a
  Removed paragraph in favour of future FAQ.md Massimo Gismondi 2025-01-18 12:41:09 +0100
• 1cc4b59dfc
  Shortened htaccess instructions Massimo Gismondi 2025-01-18 12:40:03 +0100
• 8aee2f24bb
  Fixed space in comment Massimo Gismondi 2025-01-18 12:39:07 +0100
• b455af66e7 Adding clarification about performance and code comment Massimo Gismondi 2025-01-17 21:42:08 +0100
• 189e75bbfd Adding usage instructions Massimo Gismondi 2025-01-17 21:25:23 +0100
• 933aa6159d Implementing htaccess generation Massimo Gismondi 2025-01-07 11:02:29 +0100
• b7f908e305
  Merge pull request #66 from fabianegli/patch-1 v1.23 Glyn Normington 2025-01-07 03:54:40 +0000
• ec454b71d3 Merge pull request #67 from Nightfirecat/semrushbot ai.robots.txt 2025-01-06 20:51:56 +0000
• 565dca3dc0
  Merge pull request #67 from Nightfirecat/semrushbot Cory Dransfeldt 2025-01-06 12:51:43 -0800
• 143f8f2285
  Block SemrushBot Jordan Atwood 2025-01-06 12:34:38 -0800
• 8e98cc6049
  Merge pull request #61 from glyn/improve-naming Cory Dransfeldt 2025-01-06 08:10:47 -0800
• 30ee957011
  bail when NO changes are staged Fabian Egli 2025-01-06 12:05:42 +0100
• 83cd546470
  allow Action to succeed even if no changes were made Fabian Egli 2025-01-06 11:39:41 +0100
• 0777befd36
  Create .htaccess Erdem 2025-01-06 02:56:07 +0300
• 15fc333a70
  Update robots.txt Erdem 2025-01-06 02:38:12 +0300
• ca8620e28b Merge pull request #63 from glyn/push-paths v1.22 ai.robots.txt 2025-01-05 05:05:20 +0000
• b9df958b39
  Merge pull request #63 from glyn/push-paths Glyn Normington 2025-01-05 05:05:01 +0000
• c01a684036 Convert robots.json more frequently Glyn Normington 2025-01-05 05:03:50 +0000
• d2be15447c
  Merge pull request #62 from ai-robots-txt/missing-dependency Glyn Normington 2025-01-05 01:46:27 +0000
• 9e372d0696 Ensure dependency installed missing-dependency Glyn Normington 2025-01-05 01:45:33 +0000
• 996b9c678c Improve job name Glyn Normington 2025-01-04 05:28:41 +0000
• e4c12ee2f8 Rename in test code Glyn Normington 2025-01-04 05:03:48 +0000
• 3a43714908 Rename Python code Glyn Normington 2025-01-04 04:55:34 +0000
• 2036a68c1f Update from Dark Visitors v1.21 dark-visitors 2024-12-04 00:55:50 +0000
• 24666e8b15
  Merge pull request #58 from fabianegli/fabianegli-restore-attribution v1.20 Glyn Normington 2024-11-29 09:05:16 +0000
• eb8e1a49b5 Revert "specify file encodings in tests" fabianegli 2024-11-29 09:02:47 +0100
• b64284d684 restore correct attribution logic to before PR #55 fabianegli 2024-11-26 09:41:46 +0100
• bd38c30194 specify file encodings in tests fabianegli 2024-11-26 09:12:11 +0100
• aff37825ce
  Create azure-functions-app-python.yml Tyler Hawthorne 2024-11-25 09:14:53 -0500
• db9643b29b
  Create python-publish.yml Tyler Hawthorne 2024-11-25 09:14:18 -0500
• 609ddca392 Updated from new robots.json dark-visitors 2024-11-24 00:57:06 +0000
• 37065f9118 Update from Dark Visitors dark-visitors 2024-11-24 00:57:05 +0000
• 58985737e7 Updated from new robots.json dark-visitors 2024-11-19 16:46:21 +0000
• 584e66cb99
  Merge pull request #56 from glyn/40-exclude-facebookexternalhit Cory Dransfeldt 2024-11-19 08:46:05 -0800
• 80002f5e17 Allow facebookexternalhit Glyn Normington 2024-11-19 03:33:45 +0000
• 71db599b41
  Merge pull request #55 from norwd/feature/add-robots.txt-file-to-release Glyn Normington 2024-11-13 01:39:11 +0000
• e8f0784a00
  Explicitly use release tag for checkout Y. Meyer-Norwood 2024-11-13 10:26:37 +1300
• 94ceb3cffd
  Add authentication for gh command Y. Meyer-Norwood 2024-11-11 13:04:55 +1300
• adfd4af872
  Create upload-robots-txt-file-to-release.yml Y. Meyer-Norwood 2024-11-11 12:58:40 +1300
• d50615d394 Improve formatting Glyn Normington 2024-11-10 01:06:13 +0000
• 2c88909be3 Fix formatting Glyn Normington 2024-11-10 01:02:18 +0000