cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-05-17 16:03:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Merge branch 'main' into add-traefik-plugin

Browse source
This commit is contained in:
Florent Poinsaut 2025-05-14 22:06:56 +02:00 • committed by GitHub
commit c66d180295
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 39 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

3
README.md
View file

@ -35,6 +35,9 @@ Note that, as stated in the [httpd documentation](https://httpd.apache.org/docs/
```
(Note that the path of the `haproxy-block-ai-bots.txt` may be different in your environment.)
[Bing uses the data it crawls for AI and training, you may opt out by adding a `meta` tag to the `head` of your site.](./docs/additional-steps/bing.md)
### Related
- [Robots.txt Traefik plugin](https://plugins.traefik.io/plugins/681b2f3fba3486128fc34fae/robots-txt-plugin):

36
docs/additional-steps/bing.md Normal file
View file

@ -0,0 +1,36 @@
# Bing (bingbot)
It's not well publicised, but Bing uses the data it crawls for AI and training.
However, the current thinking is, blocking a search engine of this size using `robots.txt` seems a quite drastic approach as it is second only to Google and could significantly impact your website in search results.
Additionally, Bing powers a number of search engines such as Yahoo and AOL, and its search results are also used in Duck Duck Go, amongst others.
Fortunately, Bing supports a relatively simple opt-out method, requiring an additional step.
## How to opt-out of AI training
You must add a metatag in the `<head>` of your webpage. This also needs to be added to every page on your website.
The line you need to add is:
```plaintext
<meta name="robots" content="noarchive">
```
By adding this line, you are signifying to Bing: "Do not use the content for training Microsoft's generative AI foundation models."
## Will my site be negatively affected
Simple answer, no.
The original use of "noarchive" has been retired by all search engines. Google retired its use in 2024.
The use of this metatag will not impact your site in search engines or in any other meaningful way if you add it to your page(s).
It is now solely used by a handful of crawlers, such as Bingbot and Amazonbot, to signify to them not to use your data for AI/training.
## Resources
Bing Blog AI opt-out announcement: https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat
Bing metatag information, including AI opt-out: https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240