When the World Wide Web went live in the early 1990s, its founders it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.
The has been recording the history of the internet and making it available to the public through its since . Now, some of the world’s biggest news outlets are the archive’s access to their pages.
Major publishers – including The Guardian, The New York Times, the Financial Times, and USA Today – have confirmed they’re ending the Internet Archive’s access to their content.
While they support the archive’s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls.
Yet, publishers don’t simply want to lock out AI crawlers. Rather, they want to to data-hungry tech companies. Their back catalogues of news, books and other media have become a as data to train AI systems.
Robot readers
Generative AI systems such as ChatGPT, Copilot and Gemini require access to large archives of content (such as media content, books, art and academic research) for and to .
Publishers claim technology companies have accessed a lot of this content for free and . Some began taking tech companies to court, claiming they had stolen their intellectual property. High-profile examples include ’ case against ChatGPT’s parent company OpenAI and .
Old news, new money
In response, some tech companies have to pay for access to publishers’ content. NewsCorp’s contract with OpenAI is reportedly over five years.
Similar deals have been struck between academic publishers and tech companies. Publishing houses such as have come under scrutiny in the past for locking publicly funded research behind commercial paywalls.
Now, has signed a US$10 million nonexclusive deal with Microsoft granting the company access to over 3,000 journals.
Publishers are also using accessing their content, including the crawlers used by the Internet Archive to record internet history. News publishers have referred to the Internet Archive as a “” to their catalogues, allowing unscrupulous tech companies to continue scraping their content.
The cost of making news free
The Wayback Machine has also been used by members of the public to . Understandably, media outlets want readers to pay for news.
News is a business, and its has come under increasing pressure from the same tech companies using news content for AI training and retrieval. But this comes at the expense of public access to credible information.
When newspapers first started moving their content online and making it free to the public in the late 1990s, they contributed to the ethos of sharing and collaboration on the early web.
In hindsight, however, one commentator called free access the “” of online news. The public became accustomed to getting their digital editions for free, and as online business models shifted, many mid- and small-sized news companies struggled to fund their operations.
The opposite approach – placing all commercial news behind paywalls – has its own problems. As news publishers move to , people have to juggle multiple expensive subscriptions or limit their . Otherwise, they’re left with whatever news remains online for free or is served up by social media . The result is a more closed, commercial internet.
This isn’t the first time that the Internet Archive has been in the , as the organisation was previously sued and found to be in breach of copyright through its Open Library project.
The past and future of the internet
The Wayback Machine has served as a public record of the web for , used by researchers, educators, journalists and amateur internet historians.
Blocking its access to international newspapers of note will leave significant holes in the public record of the internet.
Today, you can use the to see The New York Times’ front page from June 1997: the first time the Internet Archive crawled the newspaper’s website. In another 30 years, internet researchers and curious members of the public won’t have access to today’s front page, even if the Internet Archive is still around.
Today’s websites become tomorrow’s historical records. Without the preservation efforts of not-for-profit organisations like The Internet Archive, .
Despite the actions of commercial publishers and , not-for-profit organisations such as the Internet Archive and aim to keep the dream of an open, collaborative and transparent internet alive.
Original article
Comments