cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-16 15:27:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Include "AI Agents" from Dark Visitors

Browse source
This commit is contained in:
Joshua Sheard 2025-04-14 15:46:01 +01:00 committed by GitHub
parent 305188b2e7
commit d9f882a9b2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
code/robots.py
View file

@ -30,6 +30,7 @@ def updated_robots_json(soup):
"""Update AI scraper information with data from darkvisitors."""
existing_content = load_robots_json()
to_include = [
"AI Agents",
"AI Assistants",
"AI Data Scrapers",
"AI Search Crawlers",