cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-04-05 19:37:45 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
9 commits 2 branches 27 tags 564 KiB
Commit graph

3 commits

Author SHA1 Message Date
--Explosion--
3e57b5ab5d
Add GoogleOther 
Used by Google to crawl for internal research and development. It’s unknown what exactly this entails, but is a generic user agent that is used when no other appropriate user agent is available. Documentation available from Google: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
 2024-03-28 10:00:58 -05:00
Cory Dransfeldt
297071a664
Update robots.txt 2024-03-27 11:41:12 -07:00
Cory Dransfeldt
8aeddbdce8
Create robots.txt 2024-03-27 10:59:01 -07:00