Perplexity, an AI search startup supported by Jeff Bezos, aims to revolutionize internet usage. However, it seems that the company is potentially disregarding established web guidelines, specifically the Robots Exclusion Protocol. This protocol allows website operators to prevent access to certain parts of their website by bots. Developer Robb Knight’s report, which was validated by Wired, suggests that Perplexity is scraping web content that is intended to be restricted.
According to a blog post, Perplexity’s service provides a summary of articles on the web, promising “reliable answers” without the need to click on multiple links. Wired and Knight discovered that Perplexity bypasses code (robots.txt files) that is designed to prevent web crawlers from accessing certain websites. The reports reveal that Perplexity uses an undisclosed IP address to bypass these robots.txt files and scrape the websites regardless. Wired states that it had previously blocked Perplexity’s web crawler in 2024, but the AI search engine is still able to provide detailed summaries of its articles.
Despite this, Perplexity claims to adhere to the Robots Exclusion Protocol, as stated in their website documentation. However, Perplexity CEO Aravind Srinivas did not directly dispute the findings but instead claimed that the reporters had a misunderstanding of how Perplexity and the Internet function, according to Wired. Gizmodo contacted Perplexity for a more detailed response and will update the article if a response is received.
In addition, Perplexity is currently facing legal threats related to copyright infringement for violating other internet regulations. Forbes recently threatened legal action against Perplexity for allegedly using Forbes’ reporting without proper attribution. Forbes had conducted original reporting on the AI drone venture of former Google CEO Eric Schmidt, while Perplexity generated AI-generated articles, podcasts, and videos using Forbes’ text and images. The executive editor of Forbes previously criticized Perplexity on X earlier this month.
Perplexity’s product, while valuable, redirects internet traffic. In contrast, Google indexes webpages and provides concise AI summaries, directing traffic directly to the original sources. This poses a challenge for Perplexity as it generates detailed AI articles, reducing the need for users to visit websites and undermining the digital media business model.
To address this issue, OpenAI has established partnerships with media companies, compensating them upfront for content licenses. Perplexity is also reportedly working on similar partnerships, but with a revenue-sharing approach instead of a flat fee payment like OpenAI. However, these partnerships are not yet in place, so currently, Perplexity seems to be bypassing paywalls and scraping websites to gather the necessary information to power its AI responses.
For the latest news, users typically turn to platforms such as Facebook, Twitter, and Instagram.