CS 891 Web Archiving Seminar

100%

CS 891 Web Archiving Seminar

✓ Published0🌍 Public

Last edited Oct 4, 2017

Created on Oct 2, 2017

This visualization demonstrates the process of web archiving by showing the WARC file records produced when using `wget` to archive two websites: `http://example.com` and the React Router documentation. The code uses `wget` with extensive arguments including `--mirror`, `--warc-cdx`, and `--page-requisites` to capture complete page snapshots. The example displays the resulting WARC file structure, containing request, response, metadata, and resource records, along with a shell script that orchestrates the archiving workflow and manages the Wayback Machine collections.

AI-generated description

slides prez

Similar vizzes

Loading thumbnail…

Gist 2d272134b57b02eb8cdd0c5904249591

This example tests the archival fidelity of a web page by loading a series of images and scripts that mimic dynamic behavior, including DOM manipulation, AJAX requests, and delayed updates. It uses JavaScript’s `XMLHttpRequest` to fetch remote images and replaces them via base64 data URLs, while `setTimeout` and scroll events trigger asynchronous content changes. The page is rendered within an iframe by the pywb web archiving tool, with a banner and rewrite layer applied. The visualization demonstrates how archived sites handle live interactions and resource loading over time.

NN0taN3rd

69% match

Loading thumbnail…

2017-07-25: Swimming In A Ocean Of JavaScript. Using Nodejs, Electron And Chrome To Preserve The Web

This example demonstrates a Node.js toolkit for web archiving, showing how to parse, stream, and generate WARC and CDXJ files to preserve web content. It uses the `cdxj` library to read CDXJ indexes asynchronously, the `node-warc` parser with event listeners for streaming WARC records, and a configuration object for the `squid` crawler that controls Chrome via its WebSocket API. The code highlights handling HTTP/2 limitations in archives and supports features like page-same-domain crawling and auto-scrolling.

NN0taN3rd

68% match

Loading thumbnail…

Dump warc

This example shows a tool that extracts and saves web resources from a WARC archive file into a local directory, organizing files by their original URLs and content types. It parses the archive using the `node-warc` library in JavaScript or `warcio` in Python, mapping HTTP GET requests to their corresponding 200 OK responses. The code uses `filenamify-url` or `goldfinch` to sanitize filenames, and `fs-extra` or `pathlib` for file operations, writing each HTML, image, CSS, or JavaScript file with an appropriate file extension based on the response’s Content-Type header.

NN0taN3rd

67% match

Loading thumbnail…

Mat Kelly, PhD Student

This visualization examines the structure and contents of two WARC files from different crawls of Mat Kelly's academic homepage. It animates a series of frames that map the sequence of HTTP requests and responses, using a force-directed layout via `d3.forceSimulation` to position nodes by their relationships. The rendering combines SVG, Canvas, and WebGL to display the animation, with timing controlled by `transition.attrTween`. The data is parsed directly from the provided `heritrix.warc` and `wget.warc` files, visualizing crawl behavior and server responses like redirects and robots.txt blocks.

NN0taN3rd

67% match

Loading thumbnail…

In Archive, Alexa, Twitter Sample

This visualization presents a comparative analysis of the DOM and network complexity of seven popular websites, based on JSON snapshots captured by a web crawler. Each site is represented as a distinct point, with its position determined by aggregated metrics such as total node count, the number of clickable elements, and the diversity of resource types loaded. The tool, built with React, ingests these raw JSON files and uses a custom layout algorithm to project the multi-dimensional data onto a two-dimensional plane. Hovering over a point reveals a detailed breakdown of a site's specific DOM attributes, framework detection results, and the distribution of network requests by MIME type.

NN0taN3rd

62% match

Loading thumbnail…

WGET example

The visualization shows a network of hyperlinks and page relationships from a website crawl, using a force-directed graph layout. It employs D3.js’s `d3.forceSimulation` to position nodes (representing pages and resources) and `d3.link` to draw edges, with node size reflecting in-degree. The example is generated from a Wget command (`wget -mkEpnp http://example.org`) that mirrors a site, and the code parses the resulting HTML to extract links and build the graph. Data source is the locally mirrored site structure, rendered via SVG with transparent node fills and gray links, emphasizing connectivity patterns without animation or interaction.

HHugoberry

62% match

Loading thumbnail…

Web Archive Bookmarklet Code

A one-line JavaScript bookmarklet opens the current page in the Internet Archive’s Wayback Machine. The code triggers `window.open` on the URL formed by concatenating `'https://web.archive.org/web/'` with the live document’s `location.href`. This example demonstrates URL string construction and the Web’s `window.open` API for navigation, with no charting or graphics libraries involved. The plain text snippet contains no data visualization logic, showing instead a minimal utility for historical page capture.

NNGuernse

62% match

Loading thumbnail…

Still Drinking Gin And Juice

This example plays a trick on the Wayback Machine by loading a hidden iframe that captures the archived page’s React and ReactDOM libraries, then re-mounting a new React component inside that iframe. The visualization itself is not yet defined in the provided snippet, but the code sets up a full-screen canvas-driven animation framework using React and the browser’s native canvas and animation APIs. Images of frogs and a banner are fetched from the Internet Archive’s geocities.com captures, hinting at a playful, retro-themed visual collage. The page also manipulates browser history to disguise its origin and hides the archive’s toolbar.

NN0taN3rd

60% match