diff --git a/robots.txt b/robots.txt index 51c5e31..cf20329 100644 --- a/robots.txt +++ b/robots.txt @@ -10,6 +10,7 @@ User-agent: cohere-ai User-agent: Diffbot User-agent: FacebookBot User-agent: FriendlyCrawler +User-agent: facebookexternalhit User-agent: Google-Extended User-agent: GoogleOther User-agent: GoogleOther-Image @@ -22,6 +23,7 @@ User-agent: OAI-SearchBot User-agent: omgili User-agent: omgilibot User-agent: PerplexityBot +User-agent: PetalBot User-agent: Scrapy User-agent: Timpibot User-agent: VelenPublicWebCrawler