cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-04 19:13:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Update robots.json with Claude respect link

Browse source
This commit is contained in:
Laker Turner 2024-10-07 17:38:01 +01:00 committed by GitHub
parent 6da804e826
commit dc15afe847
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
robots.json
View file

@ -67,7 +67,7 @@
"frequency": "No information. provided.",
"function": "Scrapes data to train Anthropic's AI products.",
"operator": "[Anthropic](https://www.anthropic.com)",
"respect": "Unclear at this time."
"respect": "[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)"
},
"Diffbot": {
"description": "Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.",
@ -279,4 +279,4 @@
"operator": "[Webz.io](https://webz.io/)",
"respect": "[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)"
}
}
}