Web Archives Preserve Our Digital Collective Memory

Daniela Major,Daniel Gomes

DOI: https://doi.org/10.1007/978-3-030-63291-5_2

2021-01-01

Abstract:This chapter discusses the importance of web archiving, briefly presents its history from the beginning with the Internet Archive in 1996 and exposes the challenges with archiving certain types of online data.

What problem does this paper attempt to address?

Web archives as a data resource for digital scholars

Eveline Vlassenroot,Sally Chambers,Emmanuel Di Pretoro,Friedel Geeraert,Gerald Haesendonck,Alejandra Michel,Peter Mechant

DOI: https://doi.org/10.1007/s42803-019-00007-7

2019-03-08

International Journal of Digital Humanities

Abstract:The aim of this article is to provide an exploratory analysis of the landscape of web archiving activities in Europe. Our contribution, based on desk research, and complemented with data from interviews with representatives of European heritage institutions, provides a descriptive overview of the state-of-the-art of national web archiving in Europe. It is written for a broad interdisciplinary audience, including cultural heritage professionals, IT specialists and managers, and humanities and social science researchers. The legal, technical and operational aspects of web archiving and the value of web archives as born-digital primary research resources are both explored. In addition to investigating the organisations involved and the scope of their web archiving programmes, the curatorial aspects of the web archiving process, such as selection of web content, the tools used and the provision of access and discovery services are also considered. Furthermore, general policies related to web archiving programmes are analysed. The article concludes by offering four important issues that digital scholars should consider when using web archives as a historical data source. Whilst recognising that this study was limited to a sample of only nine web archives, this article can nevertheless offer some useful insights into the technical, legal, curatorial and policy-related aspects of web archiving. Finally, this paper could function as a stepping stone for more extensive and qualitative research.
Stories From the Past Web

Yasmin AlNoamany,Michele C. Weigle,Michael L. Nelson

DOI: https://doi.org/10.48550/arXiv.1705.06218

2017-05-17

Abstract:Archiving Web pages into themed collections is a method for ensuring these resources are available for posterity. Services such as Archive-It exists to allow institutions to develop, curate, and preserve collections of Web resources. Understanding the contents and boundaries of these archived collections is a challenge for most people, resulting in the paradox of the larger the collection, the harder it is to understand. Meanwhile, as the sheer volume of data grows on the Web, "storytelling" is becoming a popular technique in social media for selecting Web resources to support a particular narrative or "story". There are multiple stories that can be generated from an archived collection with different perspectives about the collection. For example, a user may want to see a story that is composed of the key events from a specific Web site, a story that is composed of the key events of the story regardless of the sources, or how a specific event at a specific point in time was covered by different Web sites, etc. In this paper, we provide different case studies for possible types of stories that can be extracted from a collection. We also provide the definitions and models of these types of stories.

Digital Libraries
Archival strategies for contemporary collecting in a world of big data: Challenges and opportunities with curating the UK web archive

Nicola Jayne Bingham,Helena Byrne

DOI: https://doi.org/10.1177/2053951721990409

2021-01-01

Abstract:In this contribution, we will discuss the opportunities and challenges arising from memory institutions' need to redefine their archival strategies for contemporary collecting in a world of big data. We will reflect on this topic by critically examining the case study of the UK Web Archive, which is made up of the six UK Legal Deposit Libraries: the British Library, National Library of Scotland, National Library of Wales, Bodleian Libraries Oxford, Cambridge University Library and Trinity College Dublin. The UK Web Archive aims to archive, preserve and give access to the UK web space. This is achieved through an annual domain crawl, first undertaken in 2013, in addition to more frequent crawls of key websites and specially curated collections which date back as far as 2005. These collections reflect important aspects of British culture and events that shape society. This commentary will explore a number of questions including: what heritage is captured and what heritage is instead neglected by the UK Web archive? What heritage is created in the form of new data and what are its properties? What are the ethical issues that memory institutions face when developing these web archiving practices? What transformations are required to overcome such challenges and what institutional futures can we envisage?

social sciences, interdisciplinary
Understanding Web Archiving Services and Their (Mis)Use on Social Media

Savvas Zannettou,Jeremy Blackburn,Emiliano De Cristofaro,Michael Sirivianos,Gianluca Stringhini

DOI: https://doi.org/10.48550/arXiv.1801.10396

2018-04-09

Abstract:Web archiving services play an increasingly important role in today's information ecosystem, by ensuring the continuing availability of information, or by deliberately caching content that might get deleted or removed. Among these, the Wayback Machine has been proactively archiving, since 2001, versions of a large number of Web pages, while newer services like <a class="link-external link-http" href="http://archive.is" rel="external noopener nofollow">this http URL</a> allow users to create on-demand snapshots of specific Web pages, which serve as time capsules that can be shared across the Web. In this paper, we present a large-scale analysis of Web archiving services and their use on social media, shedding light on the actors involved in this ecosystem, the content that gets archived, and how it is shared. We crawl and study: 1) 21M URLs from <a class="link-external link-http" href="http://archive.is" rel="external noopener nofollow">this http URL</a>, spanning almost two years, and 2) 356K <a class="link-external link-http" href="http://archive.is" rel="external noopener nofollow">this http URL</a> plus 391K Wayback Machine URLs that were shared on four social networks: Reddit, Twitter, Gab, and 4chan's Politically Incorrect board (/pol/) over 14 months. We observe that news and social media posts are the most common types of content archived, likely due to their perceived ephemeral and/or controversial nature. Moreover, URLs of archiving services are extensively shared on "fringe" communities within Reddit and 4chan to preserve possibly contentious content. Lastly, we find evidence of moderators nudging or even forcing users to use archives, instead of direct links, for news sources with opposing ideologies, potentially depriving them of ad revenue.

Computers and Society,Digital Libraries,Social and Information Networks
On the Change in Archivability of Websites Over Time

Mat Kelly,Justin F. Brunelle,Michele C. Weigle,Michael L. Nelson

DOI: https://doi.org/10.48550/arXiv.1307.8067

2013-07-30

Digital Libraries

Abstract:As web technologies evolve, web archivists work to keep up so that our digital history is preserved. Recent advances in web technologies have introduced client-side executed scripts that load data without a referential identifier or that require user interaction (e.g., content loading when the page has scrolled). These advances have made automating methods for capturing web pages more difficult. Because of the evolving schemes of publishing web pages along with the progressive capability of web preservation tools, the archivability of pages on the web has varied over time. In this paper we show that the archivability of a web page can be deduced from the type of page being archived, which aligns with that page's accessibility in respect to dynamic content. We show concrete examples of when these technologies were introduced by referencing mementos of pages that have persisted through a long evolution of available technologies. Identifying these reasons for the inability of these web pages to be archived in the past in respect to accessibility serves as a guide for ensuring that content that has longevity is published using good practice methods that make it available for preservation.
Full-Text and URL Search Over Web Archives

Miguel Costa

DOI: https://doi.org/10.48550/arXiv.2108.01603

2021-08-03

Digital Libraries

Abstract:Web archives are a historically valuable source of information. In some respects, web archives are the only record of the evolution of human society in the last two decades. They preserve a mix of personal and collective memories, the importance of which tends to grow as they age. However, the value of web archives depends on their users being able to search and access the information they require in efficient and effective ways. Without the possibility of exploring and exploiting the archived contents, web archives are useless. Web archive access functionalities range from basic browsing to advanced search and analytical services, accessed through user-friendly interfaces. Full-text and URL search have become the predominant and preferred forms of information discovery in web archives, fulfilling user needs and supporting search APIs that feed complex applications. Both full-text and URL search are based on the technology developed for modern web search engines, since the Web is the main resource targeted by both systems. However, while web search engines enable searching over the most recent web snapshot, web archives enable searching over multiple snapshots from the past. This means that web archives have to deal with a temporal dimension that is the cause of new challenges and opportunities, discussed throughout this chapter.
Big Data Science Over the Past Web

Miguel Costa,Julien Masanès

DOI: https://doi.org/10.1007/978-3-030-63291-5_21

2021-08-04

Abstract:Web archives preserve unique and historically valuable information. They hold a record of past events and memories published by all kinds of people, such as journalists, politicians and ordinary people who have shared their testimony and opinion on multiple subjects. As a result, researchers such as historians and sociologists have used web archives as a source of information to understand the recent past since the early days of the World Wide Web. The typical way to extract knowledge from a web archive is by using its search functionalities to find and analyse historical content. This can be a slow and superficial process when analysing complex topics, due to the huge amount of data that web archives have been preserving over time. Big data science tools can cope with this order of magnitude, enabling researchers to automatically extract meaningful knowledge from the archived data. This knowledge helps not only to explain the past but also to predict the future through the computational modelling of events and behaviours. Currently, there is an immense landscape of big data tools, machine learning frameworks and deep learning algorithms that significantly increase the scalability and performance of several computational tasks, especially over text, image and audio. Web archives have been taking advantage of this panoply of technologies to provide their users with more powerful tools to explore and exploit historical data. This chapter presents several examples of these tools and gives an overview of their application to support longitudinal studies over web archive collections.

Digital Libraries
Digital archives in the cloud: Collective memory, institutional histories and the politics of information

Michael A. Peters,Tina Besley

DOI: https://doi.org/10.1080/00131857.2018.1526669

2018-10-24

Educational Philosophy and Theory

Abstract:The archive is a cultural institution that creates a framework for the social and collective memory and as such is one of the collection of knowledge institutions that not only preserves and classifies “texts” but uses them to re-create collective memory and sometimes to invent cultural histories. Like all knowledge institutions, the archive is also a construction deeply implicated in knowledge politics or what Foucault calls power/knowledge. In the past the archive has functioned as a central metaphor for the construction of human knowledge in all it is different institutional forms and like the encyclopedia and the camera, the archive produces highly coded representations that make implicit validity claims to the truth and justice of the past. Politically speaking, those who control the archive control the past. In the digital world, the archive is used to describe a machine-readable location as a store for “data” and “information.” Digital technologies radically alter our existing institutions, making access to their embedded knowledge widely available and enable learning and research anytime, anywhere. Data analytics algorithmically can manipulate electorates and entire democracies in new ways, while destabilising the free press. This article asks what digitizing an archive means for collective memory, for the history of institutions and for politics in the Cloud.

education & educational research
Creating Structure in Web Archives With Collections: Different Concepts From Web Archivists

Himarsha R. Jayanetti,Shawn M. Jones,Martin Klein,Alex Osbourne,Paul Koerbin,Michael L. Nelson,Michele C. Weigle

DOI: https://doi.org/10.48550/arXiv.2209.08649

2022-09-19

Abstract:As web archives' holdings grow, archivists subdivide them into collections so they are easier to understand and manage. In this work, we review the collection structures of eight web archive platforms: : Archive-It, Conifer, the Croatian Web Archive (HAW), the Internet Archive's user account web archives, Library of Congress (LC), PANDORA, Trove, and the UK Web Archive (UKWA). We note a plethora of different approaches to web archive collection structures. Some web archive collections support sub-collections and some permit embargoes. Curatorial decisions may be attributed to a single organization or many. Archived web pages are known by many names: mementos, copies, captures, or snapshots. Some platforms restrict a memento to a single collection and others allow mementos to cross collections. Knowledge of collection structures has implications for many different applications and users. Visitors will need to understand how to navigate collections. Future archivists will need to understand what options are available for designing collections. Platform designers need it to know what possibilities exist. The developers of tools that consume collections need to understand collection structures so they can meet the needs of their users.

Digital Libraries
Between Lived and Archived Memory: How Digital Archives Can Tell History

Katja Müller

DOI: https://doi.org/10.7238/D.V0I19.3085

2017-01-15

Abstract:Digital archives, as they are set up online by Indian stakeholders, perform the dual task of involving practices of lived memory and storing archival information. They decidedly claim to preserve the past, and they actively engage users as prosumers on the web who interact with the creation and sharing of the digital archives’ content. Two Indian digital archives exemplify how this is done successfully. These archives are thus digital spaces that scrutinize the distinct line between lieux de memoire and milieux de memoire as a conventional concept of distinguishing history from memory. Yet, at the same time, they rely on ideas of History and memory that reinvent notions of archives as authoritarian voices.

History,Computer Science
Web archives for data collection: An ethics case study

Deanna Zarrillo,Mat Kelly,Erjia Yan,Chaoqun Ni

DOI: https://doi.org/10.1080/08989621.2024.2396940

2024-09-08

Abstract:Background: Web archives offer researchers a promising source for large-scale longitudinal data collection; however, their complex social and technical infrastructures create an array of ethical concerns. In addition, there is a notable lack of guidance available for researchers hoping to conduct0 ethical research using web archives. Methods: We present an ethical decision-making case study based on an ongoing research project using the Internet Archive's Wayback Machine to study faculty appointments and mobility at Historically Black Colleges and Universities (HBCUs). Results: This paper contributes to information ethics discourse by expanding on the Association of Internet Researchers' recommendations for ethical decision-making, and mapping ethical considerations for each stage of the project within existing conceptual frameworks for research using web archives. Conclusions: By utilizing internet research guidance and web archive research frameworks in a case study approach, we hope to aid future researchers conducting internet research of a similar nature by serving as a useful reference.
Methods and Approaches to Using Web Archives in Computational Communication Research

Matthew S. Weber

DOI: https://doi.org/10.1080/19312458.2018.1447657

IF: 8.044

2018-03-22

Communication Methods and Measures

Abstract:This article examines the role of web archives as a critical source of data for conducting computational communication research. Web archives are large-scale databases containing comprehensive records of websites showing how those websites have evolved over time. Recent communication scholarship using web archives is reviewed, demonstrating the breadth of research conducted in this space. Subsequently, a methodological framework is proposed for using web archives in computational communication research. As a source of data, web archives present a number of methodological challenges, particularly with regards to the accuracy and completeness of web archives. These problems are addressed in order to better inform future work in this area. The closing sections outline a forward-looking trajectory for computational communication research using web archives.

communication
THE PRESERVATION OF HISTORICAL MEMORY IN THE DIGITAL AGE: CHALLENGES, OPPORTUNITIES AND INTERACTIVE TECHNOLOGIES IN TEACHING AND RESEARCH

Rodger Roberto Alves de Sousa

DOI: https://doi.org/10.5216/ia.v48i3.76254

2023-12-29

Revista Inter-Ação

Abstract:This article addresses the importance of preserving historical memory in the digital age, highlighting challenges, opportunities and the use of interactive technologies in teaching and research. It discusses the ephemerality of digital media and the need for preservation strategies, such as format migration, metadata maintenance, and backup practices. The risks of loss and obsolescence of digital formats are addressed, underscoring the importance of constantly monitoring formats and adopting open standards. Legal and ethical issues related to digital preservation are examined, emphasizing respect for copyright, protection of privacy, and promotion of democratic access to historical knowledge. Global access and democratization of historical knowledge through online platforms, digital repositories and virtual libraries are also highlighted. Collaboration and information sharing between researchers and institutions are considered essential for the preservation and dissemination of historical knowledge. It presents the use of digital tools, virtual reality, games and virtual platforms in historical education, highlighting their ability to create immersive experiences for students. Big data analysis and historical data mining are explored as promising approaches to understanding historical patterns and trends. Thebenefits of data visualization and historical mapping in understanding and analyzing past events are discussed. Digital tools for the analysis of primary sources and the construction of historical narratives are presented. KEYWORDS: Media; Technology and History; Research; Memory and Teaching.
Introducing A Dark Web Archival Framework

Justin F. Brunelle,Ryan Farley,Grant Atkins,Trevor Bostic,Marites Hendrix,Zak Zebrowski

DOI: https://doi.org/10.48550/arXiv.2107.04070

2021-07-08

Digital Libraries

Abstract:We present a framework for web-scale archiving of the dark web. While commonly associated with illicit and illegal activity, the dark web provides a way to privately access web information. This is a valuable and socially beneficial tool to global citizens, such as those wishing to access information while under oppressive political regimes that work to limit information availability. However, little institutional archiving is performed on the dark web (limited to the Archive.is dark web presence, a page-at-a-time archiver). We use surface web tools, techniques, and procedures (TTPs) and adapt them for archiving the dark web. We demonstrate the viability of our framework in a proof-of-concept and narrowly scoped prototype, implemented with the following lightly adapted open source tools: the Brozzler crawler for capture, WARC file for storage, and pywb for replay. Using these tools, we demonstrate the viability of modified surface web archiving TTPs for archiving the dark web.
The Long Term Fate of Our Digital Belongings: Toward a Service Model for Personal Archives

Catherine C. Marshall,Sara Bly,Francoise Brun-Cottan

DOI: https://doi.org/10.48550/arXiv.0704.3653

2007-04-27

Abstract:We conducted a preliminary field study to understand the current state of personal digital archiving in practice. Our aim is to design a service for the long-term storage, preservation, and access of digital belongings by examining how personal archiving needs intersect with existing and emerging archiving technologies, best practices, and policies. Our findings not only confirmed that experienced home computer users are creating, receiving, and finding an increasing number of digital belongings, but also that they have already lost irreplaceable digital artifacts such as photos, creative efforts, and records. Although participants reported strategies such as backup and file replication for digital safekeeping, they were seldom able to implement them consistently. Four central archiving themes emerged from the data: (1) people find it difficult to evaluate the worth of accumulated materials; (2) personal storage is highly distributed both on- and offline; (3) people are experiencing magnified curatorial problems associated with managing files in the aggregate, creating appropriate metadata, and migrating materials to maintainable formats; and (4) facilities for long-term access are not supported by the current desktop metaphor. Four environmental factors further complicate archiving in consumer settings: the pervasive influence of malware; consumer reliance on ad hoc IT providers; an accretion of minor system and registry inconsistencies; and strong consumer beliefs about the incorruptibility of digital forms, the reliability of digital technologies, and the social vulnerability of networked storage.

Digital Libraries,Computers and Society,Human-Computer Interaction
Preserving the Past, Enabling the Future: Assessing the European Policy on Access to Archives in the Digital Age

Michael Friedewald,Iván Székely,Murat Karaboga

DOI: https://doi.org/10.1515/pdtc-2024-0003

2024-04-04

Preservation, Digital Technology & Culture (PDT&C)

Abstract:In this study we investigate the implementation and challenges of accessing archives in the digital age. Through an online survey and expert interviews, data were collected from archival institutions, professional users, and civil society organizations in all Council of Europe member states. The findings reveal that digitization has had a positive impact on document accessibility, providing convenience, reducing workload, and improving user access. However, only a small portion of archival holdings have been digitized, highlighting the need for further efforts in this area. Data protection and copyright present significant obstacles, with the European General Data Protection Regulation leading to access restrictions and legal uncertainties. Preservation, interoperability, and organization of records pose challenges in the digital realm, and while artificial intelligence shows promise in automating metadata creation and identifying sensitive materials, concerns about algorithmic biases and the need for additional checks are highlighted. To enhance accessibility, user-friendly interfaces, cooperation among memory preserving institution, and outreach to new user groups are crucial, which can ensure the effective preservation and accessibility of digital collections in the digital age. We provide insights for policy-making and archival institutions in navigating the evolving landscape of accessibility.
The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives

Nick Ruest,Jimmy Lin,Ian Milligan,Samantha Fritz

DOI: https://doi.org/10.48550/arXiv.2001.05399

2020-01-16

Abstract:The Archives Unleashed project aims to improve scholarly access to web archives through a multi-pronged strategy involving tool creation, process modeling, and community building - all proceeding concurrently in mutually-reinforcing efforts. As we near the end of our initially-conceived three-year project, we report on our progress and share lessons learned along the way. The main contribution articulated in this paper is a process model that decomposes scholarly inquiries into four main activities: filter, extract, aggregate, and visualize. Based on the insight that these activities can be disaggregated across time, space, and tools, it is possible to generate "derivative products", using our Archives Unleashed Toolkit, that serve as useful starting points for scholarly inquiry. Scholars can download these products from the Archives Unleashed Cloud and manipulate them just like any other dataset, thus providing access to web archives without requiring any specialized knowledge. Over the past few years, our platform has processed over a thousand different collections from about two hundred users, totaling over 280 terabytes of web archives.

Digital Libraries,Information Retrieval
Digital humanities and web archives: Possible new paths for combining datasets

Niels Brügger

DOI: https://doi.org/10.1007/s42803-021-00038-z

2021-05-28

International Journal of Digital Humanities

Abstract:This article discusses the importance of web archives making their collections available as data and not only as sources seen through the Wayback Machine’s interface where only individual web pages are displayed. This will help unlock the full potential of the treasure trove that web archives constitute, and thereby also open up for methods from the wider field of digital humanities. Based on a case study of the entire Danish web domain .dk the article discusses methodological challenges involved in combining large extracted datasets from web archives, namely metadata about the size of websites and data about hyperlinks from the same websites. The aim is to answer the following two questions: 1) How to combine two different types of datasets extracted from a web archive, in this case the Danish Netarkivet? 2) What can the result of such a combination teach us about the structural characteristics of the Danish web domain from 2006 to 2015? The article shows that, indeed, it is possible to go beyond the Wayback Machine as the prime interface to web archives by combining two distinct datasets, and that such a venture can provide valuable knowledge about the overall structure of the Danish web domain, thus highlighting that websites of the same size tend to constitute isolated ‘link islands’, and that big websites are also the most important in the hyperlink network, which is more clearly the case in 2015 than in 2006.
ArchiveWeb: collaboratively extending and exploring web archive collections - How would you like to work with your collections?

Zeon Trevor Fernando,Ivana Marenzi,Wolfgang Nejdl

DOI: https://doi.org/10.48550/arXiv.1702.00436

2017-02-01

Digital Libraries

Abstract:Curated web archive collections contain focused digital content which is collected by archiving organizations, groups, and individuals to provide a representative sample covering specific topics and events to preserve them for future exploration and analysis. In this paper, we discuss how to best support collaborative construction and exploration of these collections through the ArchiveWeb system. ArchiveWeb has been developed using an iterative evaluation-driven design-based research approach, with considerable user feedback at all stages. The first part of this paper describes the important insights we gained from our initial requirements engineering phase during the first year of the project and the main functionalities of the current ArchiveWeb system for searching, constructing, exploring, and discussing web archive collections. The second part summarizes the feedback we received on this version from archiving organizations and libraries, as well as our corresponding plans for improving and extending the system for the next release.
Archiving and Preserving Social Media at the Library of Congress: Institutional and Cultural Challenges to Build a Twitter Archive

Elisabeth Fondren,Meghan Menard McCune

DOI: https://doi.org/10.1515/pdtc-2018-0011

2018-07-26

Abstract:Abstract Institutional archiving of media is neither new nor strange. The United States Library of Congress has been preserving printed materials, newspapers, photographs, film, and even websites for decades—if not centuries. After seven years, in later 2017, the initiative to build a Twitter Archive came to a halt. Through a textual analysis of policy papers, preservation theories and press releases, this study illustrates the social, cultural, and symbolic challenges of institutional archiving of digital media.

Web Archives Preserve Our Digital Collective Memory

Web archives as a data resource for digital scholars

Stories From the Past Web

Archival strategies for contemporary collecting in a world of big data: Challenges and opportunities with curating the UK web archive

Understanding Web Archiving Services and Their (Mis)Use on Social Media

On the Change in Archivability of Websites Over Time

Full-Text and URL Search Over Web Archives

Big Data Science Over the Past Web

Digital archives in the cloud: Collective memory, institutional histories and the politics of information

Creating Structure in Web Archives With Collections: Different Concepts From Web Archivists

Between Lived and Archived Memory: How Digital Archives Can Tell History

Web archives for data collection: An ethics case study

Methods and Approaches to Using Web Archives in Computational Communication Research

THE PRESERVATION OF HISTORICAL MEMORY IN THE DIGITAL AGE: CHALLENGES, OPPORTUNITIES AND INTERACTIVE TECHNOLOGIES IN TEACHING AND RESEARCH

Introducing A Dark Web Archival Framework

The Long Term Fate of Our Digital Belongings: Toward a Service Model for Personal Archives

Preserving the Past, Enabling the Future: Assessing the European Policy on Access to Archives in the Digital Age

The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives

Digital humanities and web archives: Possible new paths for combining datasets

ArchiveWeb: collaboratively extending and exploring web archive collections - How would you like to work with your collections?

Archiving and Preserving Social Media at the Library of Congress: Institutional and Cultural Challenges to Build a Twitter Archive