Seattle, Washington – Amazon Web Services is currently investigating allegations against Perplexity AI for potential violations of its rules. The cloud division of the company is looking into claims that Perplexity AI is using a crawler hosted on its servers, which is bypassing the Robots Exclusion Protocol.
Developed in the ’90s, the Robots Exclusion Protocol is a web standard where developers use a robots.txt file to control bot access to specific web pages. While compliance is voluntary, reputable companies generally adhere to these instructions. Perplexity AI has been accused of bypassing these instructions, leading to an investigation by Amazon Web Services.
Reports indicate that Perplexity AI’s crawler was found to be scraping content from various publications like The Guardian, Forbes, and The New York Times. Despite denying any wrongdoing, Perplexity AI admitted that their chatbot would ignore robots.txt instructions if provided with a specific URL.
Amazon Web Services stated that they prohibit abusive and illegal activities, with customers responsible for complying with these terms. The company mentioned that they are reviewing the information provided by Wired, the publication that initially raised the concerns, as part of their investigation process.
CEO of Perplexity, Aravind Srinivas, has denied allegations that the company is disregarding the Robots Exclusion Protocol. However, Srinivas acknowledged that the bot identified by Wired was one of the third-party web crawlers used by the company.
In response to the investigation, Perplexity spokesperson, Sara Platnick, mentioned that their PerplexityBot respects robots.txt instructions and does not violate AWS Terms of Service. Amazon Web Services has engaged in an investigation following Wired’s report, with Perplexity stating they have responded to Amazon’s inquiries.
The situation underscores the ongoing challenges companies face with AI technology and the importance of respecting web standards to ensure fair and ethical use of data. Controversies like this highlight the need for clear guidelines and enforcement mechanisms to prevent misuse of technology in the future.