San Francisco, CA – Two leading AI startups, OpenAI and Anthropic, are under scrutiny for allegedly disregarding requests from media publishers to halt scraping their web content for AI training data. The companies have reportedly been found to be skirting the established web rule known as robots.txt, which prohibits automated scraping of websites.
TollBit, a startup facilitating paid licensing agreements between publishers and AI firms, discovered that multiple AI companies, including OpenAI and Anthropic, have been either bypassing or ignoring robots.txt. While both companies claim to respect the rule and any blocks to their specific web crawlers, it appears that these blocks are not being adhered to, with the AI companies opting to bypass robots.txt to extract content from various websites.
Despite allegations of non-compliance, OpenAI and Anthropic have not responded to requests for comment. The robots.txt protocol has been utilized by websites since the late 1990s to communicate to bot crawlers that their data should not be scraped or collected. However, the demand for high-quality training data by AI startups and tech companies has led to the undermining of this rule, along with other unofficial agreements supporting its use.
OpenAI is the developer of the popular ChatGPT chatbot, with Microsoft as its largest investor. Meanwhile, Anthropic is behind the Claude chatbot, with Amazon as its primary investor. Both chatbots are known for providing human-like responses to user inquiries, a feat made possible by the vast amounts of data scraped from the web, much of which is under copyright.
Last year, several tech firms argued to the US Copyright Office that web content should not be subject to copyright when used as AI training data. OpenAI has secured agreements with publishers like Axel Springer for content access. The US Copyright Office is expected to update its guidelines on AI and copyright later this year.
The controversy raises questions about the ethics of AI training practices and the implications for creators and publishers. As the AI industry continues to grow and innovate, finding a balance between data access and copyright protection will be crucial moving forward.