Creative Commons Backs ‘Pay-to-Crawl’ as Publishers Seek AI Revenue

Article Summary

Creative Commons, traditionally known for its work around open licensing, is now backing ‘pay-to-crawl’ systems – a controversial approach designed to address the significant disruption AI is causing to the media landscape. As AI chatbots increasingly replace traditional search traffic for publishers, the nonprofit is recognizing the urgent need for new revenue models. The ‘pay-to-crawl’ system would involve AI web crawlers paying websites a fee each time they access content for training large language models. This follows a trend already underway, with companies like Cloudflare and others developing similar technology. The model acknowledges the shift in user behavior: consumers no longer typically click through to the original source after receiving an answer from an AI. Numerous high-profile deals have emerged between AI companies and major publishers like Condé Nast, Axel Springer, and Gannett, demonstrating the scale of this transformation. However, Creative Commons is urging for responsible implementation, emphasizing principles like throttling access rather than outright blocking, ensuring public interest access, and promoting open, interoperable systems. The move is a recognition that the traditional web ecosystem is fundamentally changing, and that new mechanisms are desperately needed to ensure the viability of publishers and the continued accessibility of information.

Key Points

• Creative Commons is supporting ‘pay-to-crawl’ systems to address lost revenue for publishers due to AI-driven chatbot usage.
• The ‘pay-to-crawl’ model involves AI web crawlers paying websites for accessing their content, mirroring emerging deals between AI companies and publishers.
• Creative Commons is advocating for responsible implementation, including principles like throttling access and preserving public interest access.