cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-04 19:13:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Update README.md

Browse source 
Co-authored-by: Glyn Normington <work@underlap.org>
This commit is contained in:
Massimo Gismondi 2025-01-20 06:28:32 +01:00 committed by GitHub
parent 52241bdca6
commit 33c38ee70b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 3 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -10,10 +10,12 @@ If you'd like to add information about a crawler to the list, please make a pull
## Usage ## Usage
Many visitors will find these files from this repository most useful: This repository provides the following files:
- `robots.txt` - `robots.txt`
- `.htaccess` - `.htaccess`
### `robots.txt`
`robots.txt` implements the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)). `robots.txt` implements the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)).
### `.htaccess` ### `.htaccess`