Perplexity Caught Using Stealth Crawlers to Ignore Website Rules

Dopious

Senior Member
Founding Member
Sapphire Member
Patron
Bronze Star Bronze Star Bronze Star Bronze Star Bronze Star
Joined
Apr 5, 2025
Messages
1,431
Reaction Score
4,362
Feedback
4 / 0 / 0
Cloudflare has discovered that Perplexity AI is using undeclared web crawlers to scrape websites. These crawlers are specifically designed to ignore `robots.txt` directives, which are the standard rules website owners use to block bots.

To avoid being blocked, the crawlers disguise their identity by using generic user-agent strings, making them appear as regular browser traffic. This behavior directly contradicts Perplexity's public claims that they respect the choices of content creators and honor `robots.txt` files. Ultimately, this practice undermines the ability of website owners to control how their content is used by AI companies.

Source: https://blog.cloudflare.com/perplex...rawlers-to-evade-website-no-crawl-directives/

BlackHat all the way baby.
 
Yeah, that doesn’t surprise me. Some of these companies keep doing this, and there are other forum owners who mention it on the Xenforo forums.
 
Annoying, but I don't think I'd have acted differently if I were in their place.
 
Back
Top