cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-04 11:03:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Allow facebookexternalhit

Browse source 
At the time of writing, this crawler does not
appear to be for the purpose of AI.

See: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
(accessed on 19 November 2024).

Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/40
This commit is contained in:
Glyn Normington 2024-11-19 03:33:45 +00:00
parent 71db599b41
commit 80002f5e17
1 changed files with 0 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
robots.json
View file

@ -104,13 +104,6 @@
"frequency": "Up to 1 page per second",
"description": "Officially used for training Meta \"speech recognition technology,\" unknown if used to train Meta AI specifically."
},
"facebookexternalhit": {
"description": "Unclear at this time.",
"frequency": "Unclear at this time.",
"function": "No information.",
"operator": "Meta/Facebook",
"respect": "[Yes](https://developers.facebook.com/docs/sharing/bot/)"
},
"FriendlyCrawler": {
"description": "Unclear who the operator is; but data is used for training/machine learning.",
"frequency": "Unclear at this time.",