cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-04 19:13:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

chore: update faq

Browse source
This commit is contained in:
Cory Dransfeldt 2024-08-14 09:21:26 -07:00 committed by GitHub
parent df5b6ef647
commit bc66d10afd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
FAQ.md
View file

@ -14,6 +14,8 @@ Also, given the contentious nature of AI and the possibility of legislation limi
Yes, provided the crawlers identify themselves and your application/hosting supports doing so.
Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block.
## What can we do if a bot doesn't respect `robots.txt`?
That depends on your stack.