cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-19 16:53:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Added Test and Docu

Browse source
This commit is contained in:
Julian Beittel 2025-04-23 09:41:18 +02:00
parent e519f9f94c
commit 1e92364c13
5 changed files with 19 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -14,6 +14,7 @@ This repository provides the following files:
- `robots.txt`
- `.htaccess`
- `nginx-block-ai-bots.conf`
- `Caddyfile`
`robots.txt` implements the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)).
@ -22,6 +23,7 @@ Note that, as stated in the [httpd documentation](https://httpd.apache.org/docs/
`nginx-block-ai-bots.conf` implements a Nginx configuration snippet that can be included in any virtual host `server {}` block via the `include` directive.
`Caddyfile` includes a Header Regex matcher group you can copy or import into your Caddyfile, the rejection can then be handeld as followed `abort @aibots`
## Contributing