Web Archiving

The aim of this article is to provide a basic introduction to web archiving and highlight some of the paths you can take if you want to start archiving web content from your organization or company. Additionally, various tools for web archiving will be presented, which are accessible without requiring extensive prior technical knowledge.

Finally, three concrete cases for web archiving will be presented.

The Value of Web Archiving

Web archiving is the process where content from the internet is collected and preserved in an archive format, thus enabling future display and use.

There can be several tangible values for web archiving. From a historical perspective, there are good incentives for web archiving, as significant parts of contemporary companies’ and organizations’ internal and external communication and marketing occur via the internet. It is largely through companies’ and organizations’ presence and activities on the internet that their contemporary development can be documented, and from where aspects such as employee culture, values, and self-understanding can be clarified.

Web archiving can also have business value. Due to the internet’s constant technological development and volatility, content quickly disappears. Through web archiving, you can increase the yield of the investment made in developing content for the internet, whether in the form of web pages, social media posts, or graphics and videos. Web archiving also ensures the possibility of the content’s future use, for example, as reference and inspiration for new marketing campaigns, as a source basis for retrospective analyses, documentation in connection with legal cases, or entirely different purposes.

External Web Archives – Are Others Already Doing the Work for You?

Before seriously embarking on your own web archiving efforts, it can be advantageous to investigate whether you can find relevant content in external web archives. There is a good chance that your company’s or organization’s public web pages have already been archived – potentially multiple times over a longer period.

Internet Archive is an American nonprofit organization that has been continuously archiving billions of web pages since 1996, which can be accessed for free through their service Wayback Machine.

However, you should be aware that there may be gaps in Internet Archive’s collections, and therefore some of the web pages in their archive may appear incomplete with missing web objects (e.g., images) and dead links. It is also worth considering that, although Internet Archive’s collection is continuous, it may take longer between their archiving sessions, and in the interim, a web page may potentially have changed several times. Additionally, some web pages automatically request that Internet Archive does not archive their content.

In addition to Internet Archive, there are also national web archives, including Netarkivet, managed by the Royal Danish Library, which requires researcher access to use.

To ensure that your web content is archived, it is therefore a good idea to be proactive and not rely on others to do the work.

Although web archiving is a technical discipline, there are also tools and services that do not require extensive prior knowledge to use

Web Crawlers and WARC

For web archiving, where complete web pages are collected and preserved, programs called web crawlers are typically used. For example, Internet Archive uses the web crawler Heritrix for their work.

The process begins by assigning the web crawler a URL or domain that serves as the starting point (also called the seed) for its harvesting of content and objects. From there, the web crawler will systematically and automatically harvest the content of the web page and gradually branch out to the web pages that the seed links to, from where the harvesting continues. The web crawler’s harvesting can be limited by specifying levels that it should move in relation to internal and external links, and it can also be configured to ignore certain file types.

Typically, web crawlers can consolidate the harvested content into WARC, which is an ISO-standardized (ISO 28500:2017) archive format for web content. WARC also contains metadata that documents the technical details of the harvesting process.

Tools and Services

A web crawler such as the aforementioned Heritrix, which can harvest large amounts of web content at a highly automated level, often requires some technical insight to install, configure, and operate. However, it is not necessarily the case that you need to operate your own in-house web crawler to get started with your own web archiving – as with any other technology choice, it depends on needs, resources, and goals.

Below are examples of useful tools for your further work with web archiving.

Web crawlers with graphical user interfaces:

Easy-to-use and flexible web archiving tool that is less automated, as it requires you to navigate the web pages you want to preserve:

Commercial services that offer on-demand web crawling, organizing, displaying, and exporting of archived web content:

Can be used to take a complete screenshot of a given web page:

Cases for Web Archiving

1. A company wants to document that they comply with legal requirements by making product manuals available on their public website.

In combination with other measures, an automated process is set up where the company’s website and the relevant product manuals are continuously harvested and archived in WARC format, which is then automatically incorporated into the company’s record management.

Lack of valid documentation that can be quickly retrieved can, in this case, have negative consequences for the company’s business and reputation, whether in the form of lawsuits, fines, or customer complaints.

2. In connection with the launch of a new product, a company needs to continuously archive their communication and marketing activities for later use.

Since it is a limited amount of content, the company decides to use a selective web archiving approach, where only web content that is directly relevant to the current product launch is manually selected and harvested.

Without continuous web archiving, you risk losing web content that can be expensive to recreate. Additionally, without authentic data, you will be at a disadvantage if you later want to conduct retrospective analyses of, for example, the progress and effectiveness of the product launch.

3. An organization wants to document their actions during the COVID-19 pandemic for historical reasons.

As part of the documentation process, web content from both internal and external sources is harvested and archived.