cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-19 00:26:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge pull request #105 from jsheard/patch-1
Some checks are pending
/ run-tests (push) Waiting to run
Details

Browse source 
Include "AI Agents" from Dark Visitors
This commit is contained in:
Cory Dransfeldt 2025-04-14 10:08:38 -07:00 • committed by GitHub
commit 156e6baa09
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
code/robots.py
View file

@ -30,6 +30,7 @@ def updated_robots_json(soup):
"""Update AI scraper information with data from darkvisitors.""" """Update AI scraper information with data from darkvisitors."""
existing_content = load_robots_json() existing_content = load_robots_json()
to_include = [ to_include = [
"AI Agents",
"AI Assistants", "AI Assistants",
"AI Data Scrapers", "AI Data Scrapers",
"AI Search Crawlers", "AI Search Crawlers",