How the Wayback Machine is fighting linkrot

Key takeaways

Digital decay is a growing concern, with significant portions of the web, including valuable journalistic content, disappearing.
The Internet Archive, via its Wayback Machine, plays a crucial role in preserving the history of the web and providing access to archived content.
Challenges to digital preservation include the rise of mobile apps, hyper-personalization, and the increased closure of the internet due to geopolitical factors.
New legal challenges around web scraping for AI purposes are leading to debates on the future of web accessibility and preservation.

Summary

The Perils of Digital Decay

The episode discusses the alarming problem of 'digital decay,' where significant portions of the web are vanishing. A Pew Research study revealed that about 38% of links from 2013 are no longer accessible, equating to over a third of digital content from just a decade prior. This decay is not limited to older domains but also affects established journalism and media outlets like MTV News and Game Informer, which have recently ceased to exist. Many outlets have failed in the platform era, resulting in countless meaningful articles and archived media simply disappearing.

Preserving the Past: The Internet Archive and Wayback Machine

Mark Graham, Director of the Wayback Machine at the Internet Archive, explains the organization's mission to preserve digital content. Founded in 1996, the Internet Archive has been archiving much of the public web and has created the Wayback Machine, allowing users to view snapshots of websites. The Internet Archive's efforts include digitizing over 4,000 books daily and preserving various media types, emphasizing the importance of protecting cultural heritage as parts of the web continue to disappear.

Challenges of Digital Preservation

The conversation casts light on the challenges faced by the Internet Archive in its archiving efforts. These challenges stem not only from the ephemeral nature of the web but also from the increasing 'splinterization' of the internet where geopolitical boundaries limit access. Graham notes the motion towards a mobile-first environment filled with apps that create closed systems, complicating the archiving process. Moreover, he highlights how hyper-personalization and issues surrounding rights and privacy factor into their archiving decisions, showing that respecting privacy while preserving history is a delicate balancing act.

Emerging Threats to Open Access

Additionally, the episode highlights a recent shift in attitudes towards web scraping driven by AI companies. Many organizations are now putting restrictions on their data, questioning the ethicality of automated scraping practices. The Internet Archive works to respect guidelines laid out in 'robots.txt' files, ensuring their practices align with web standards. Graham emphasizes the necessity for transparent and cooperative relationships with platforms in maintaining access to archived content.

Show notes

The web has a problem: huge chunks of it keep going offline. The web isn’t static, parts of it sometimes just… vanish.

But it’s not all grim. The Internet Archive has a massive mission to identify and back up our online world into a vast digital library. In 2001, it launched the Wayback Machine, an interface that lets anyone call up snapshots of sites and look at how they used to be and what they used to say at a given moment in time. Mark Graham, director of the Wayback Machine, joins Decoder this week to explain both why and how the organization tries to keep the web from disappearing.

Links:

When Online Content Disappears | Pew Research

Game Informer is shutting down | The Verge

When Media Outlets Shutter, Why Are the Websites Wiped, Too? Slate

MTV News lives on in the Internet Archive | The Verge

The video game industry is mourning the loss of Game Informer | The Verge

Guest host Hank Green makes Nilay Patel explain why websites have a future | Decoder

How The Onion is saving itself from the digital media death spiral | Decoder

The Internet Archive is defending its digital library in court today | The Verge

The Internet Archive has lost its first fight to scan and lend ebooks | The Verge

The Internet Archive just lost its appeal over ebook lending | The Verge

Credits:
Decoder is a production of The Verge and part of the Vox Media Podcast Network.
Our producers are Kate Cox and Nick Statt. Our editor is Callie Wright. Our supervising producer is Liam James.
The Decoder music is by Breakmaster Cylinder.
Learn more about your ad choices. Visit podcastchoices.com/adchoices