Skip to main content
SimilarExport ZIP
100%

2017-07-25: Swimming In A Ocean Of JavaScript. Using Nodejs, Electron And Chrome To Preserve The Web

✓ Published0🌍 Public
NN0taN3rd
Last edited Jul 23, 2017
Created on Jul 22, 2017

This example demonstrates a Node.js toolkit for web archiving, showing how to parse, stream, and generate WARC and CDXJ files to preserve web content. It uses the `cdxj` library to read CDXJ indexes asynchronously, the `node-warc` parser with event listeners for streaming WARC records, and a configuration object for the `squid` crawler that controls Chrome via its WebSocket API. The code highlights handling HTTP/2 limitations in archives and supports features like page-same-domain crawling and auto-scrolling.

AI-generated description

Similar vizzes

Loading thumbnail…

CS 891 Web Archiving Seminar

This visualization demonstrates the process of web archiving by showing the WARC file records produced when using `wget` to archive two websites: `http://example.com` and the React Router documentation. The code uses `wget` with extensive arguments including `--mirror`, `--warc-cdx`, and `--page-requisites` to capture complete page snapshots. The example displays the resulting WARC file structure, containing request, response, metadata, and resource records, along with a shell script that orchestrates the archiving workflow and manages the Wayback Machine collections.

NN0taN3rd
68% match
Loading thumbnail…

Gist 2d272134b57b02eb8cdd0c5904249591

This example tests the archival fidelity of a web page by loading a series of images and scripts that mimic dynamic behavior, including DOM manipulation, AJAX requests, and delayed updates. It uses JavaScript’s `XMLHttpRequest` to fetch remote images and replaces them via base64 data URLs, while `setTimeout` and scroll events trigger asynchronous content changes. The page is rendered within an iframe by the pywb web archiving tool, with a banner and rewrite layer applied. The visualization demonstrates how archived sites handle live interactions and resource loading over time.

NN0taN3rd
66% match
Loading thumbnail…

use chrome to crawl

This example demonstrates automated web crawling and interaction using Chrome's remote interface. The code navigates through URLs from a seed list, captures page HTML and window environment variables, and programmatically clicks all elements with click event listeners. It manages JavaScript dialogs and navigation timeouts, while logging errors. It relies on Chrome's Remote Interface (CDP), along with Bluebird for promise handling, and custom detectors for JS library identification.

NN0taN3rd
63% match
Loading thumbnail…

Dump warc

This example shows a tool that extracts and saves web resources from a WARC archive file into a local directory, organizing files by their original URLs and content types. It parses the archive using the `node-warc` library in JavaScript or `warcio` in Python, mapping HTTP GET requests to their corresponding 200 OK responses. The code uses `filenamify-url` or `goldfinch` to sanitize filenames, and `fs-extra` or `pathlib` for file operations, writing each HTML, image, CSS, or JavaScript file with an appropriate file extension based on the response’s Content-Type header.

NN0taN3rd
62% match
Loading thumbnail…

Still Drinking Gin And Juice

This example plays a trick on the Wayback Machine by loading a hidden iframe that captures the archived page’s React and ReactDOM libraries, then re-mounting a new React component inside that iframe. The visualization itself is not yet defined in the provided snippet, but the code sets up a full-screen canvas-driven animation framework using React and the browser’s native canvas and animation APIs. Images of frogs and a banner are fetched from the Internet Archive’s geocities.com captures, hinting at a playful, retro-themed visual collage. The page also manipulates browser history to disguise its origin and hides the archive’s toolbar.

NN0taN3rd
60% match
Loading thumbnail…

Gist 2559ad6fd17831c9259a8af6f38ec027

This example filters a Common Crawl index file to identify archived web pages. It reads a CDXJ file using the `read_cdxj.py` script, which parses each line with the `CDXObject` class from `pywb.warcserver.index.cdxobject`. For each record, it checks whether the MIME type contains "html" and the HTTP status code equals 200, then prints the corresponding URL. The visualization shows which URLs meet these criteria, highlighting successful HTML captures from the crawl data.

NN0taN3rd
58% match
Loading thumbnail…

Usage

This example demonstrates how to replay a historical archived version of the CNN homepage through a headless Chrome browser. It loads a specific Wayback Machine capture from December 2017, setting the viewport to a fixed 1920×1080 resolution. The script uses Puppeteer to launch the browser, and injects a custom `magic` function that overrides `document.domain` to mimic the live site’s domain for proper page behavior. The navigation targets the archived URL directly, with a 60-second timeout, and any errors are logged to the console.

NN0taN3rd
58% match
Loading thumbnail…

Mat Kelly, PhD Student

This visualization examines the structure and contents of two WARC files from different crawls of Mat Kelly's academic homepage. It animates a series of frames that map the sequence of HTTP requests and responses, using a force-directed layout via `d3.forceSimulation` to position nodes by their relationships. The rendering combines SVG, Canvas, and WebGL to display the animation, with timing controlled by `transition.attrTween`. The data is parsed directly from the provided `heritrix.warc` and `wget.warc` files, visualizing crawl behavior and server responses like redirects and robots.txt blocks.

NN0taN3rd
58% match