cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-03 19:03:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Removed paragraph in favour of future FAQ.md

Browse source 
Co-authored-by: Glyn Normington <work@underlap.org>
This commit is contained in:
Massimo Gismondi 2025-01-18 12:41:09 +01:00 committed by GitHub
parent 1cc4b59dfc
commit d65128d10a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 0 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -20,7 +20,6 @@ The second one tells your own webserver to return an error page when one of the
Note that, as stated in the [httpd documentation](https://httpd.apache.org/docs/current/howto/htaccess.html), more performant methods than an `.htaccess` file exist. Nevertheless, most shared hosting providers only allow `.htaccess` configuration.
We suggest adding both files, as some crawlers may respect `robots.txt` while not having an identifiable User Agent; on the other hand, other crawlers may not respect the `robots.txt`, but they provide a identifiable User Agent by which we can filter them out.
## Contributing