AI site Perplexity uses “stealth tactics” to flout no-crawl edicts, Cloudflare says
Cloudflare has accused AI search engine Perplexity of using stealth bots to bypass websites' no-crawl directives, violating long-standing Internet norms. Despite efforts by website owners to block Perplexity scraping bots through robots.txt files and firewalls, Cloudflare claims Perplexity still accessed site content. Cloudflare researchers conducted tests and found that Perplexity employed stealth tactics to evade blocks, continuing to scrape content undetected. This situation raises concerns about the challenges posed by advanced AI technologies in circumventing established web protocols.
Ars Technica•