News sites are locking out the Internet Archive to stop AI crawling. Is the ‘open web’ closing?

Description

When the World Wide Web went live in the early 1990s, its founders hoped it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.

The Internet Archive has been recording the history of the internet and making it available to the public through its Wayback Machine since 1996. Now, some the world’s biggest news outlets are blocking the archive’s access to their pages.

Major publishers – including The Guardian, The New York Times, the Financial Times, and USA Today – have confirmed they’re ending the Internet Archive’s access to their content.

While publishers say they support the archive’s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls.

Yet, publishers don’t simply want to lock out AI crawlers. Rather, they want to sell their content to data-hungry tech companies. Their back catalogues of news, books and other media have become a hot commodity as data to train AI systems.