Dump warc

100%

✓ Published0🌍 Public

Last edited Jun 18, 2018

Created on Jun 18, 2018

This example shows a tool that extracts and saves web resources from a WARC archive file into a local directory, organizing files by their original URLs and content types. It parses the archive using the `node-warc` library in JavaScript or `warcio` in Python, mapping HTTP GET requests to their corresponding 200 OK responses. The code uses `filenamify-url` or `goldfinch` to sanitize filenames, and `fs-extra` or `pathlib` for file operations, writing each HTML, image, CSS, or JavaScript file with an appropriate file extension based on the response’s Content-Type header.

AI-generated description

Similar vizzes

Loading thumbnail…

CS 891 Web Archiving Seminar

This visualization demonstrates the process of web archiving by showing the WARC file records produced when using `wget` to archive two websites: `http://example.com` and the React Router documentation. The code uses `wget` with extensive arguments including `--mirror`, `--warc-cdx`, and `--page-requisites` to capture complete page snapshots. The example displays the resulting WARC file structure, containing request, response, metadata, and resource records, along with a shell script that orchestrates the archiving workflow and manages the Wayback Machine collections.

NN0taN3rd

67% match

Loading thumbnail…

2017-07-25: Swimming In A Ocean Of JavaScript. Using Nodejs, Electron And Chrome To Preserve The Web

This example demonstrates a Node.js toolkit for web archiving, showing how to parse, stream, and generate WARC and CDXJ files to preserve web content. It uses the `cdxj` library to read CDXJ indexes asynchronously, the `node-warc` parser with event listeners for streaming WARC records, and a configuration object for the `squid` crawler that controls Chrome via its WebSocket API. The code highlights handling HTTP/2 limitations in archives and supports features like page-same-domain crawling and auto-scrolling.

NN0taN3rd

62% match

Loading thumbnail…

Gist 2559ad6fd17831c9259a8af6f38ec027

This example filters a Common Crawl index file to identify archived web pages. It reads a CDXJ file using the `read_cdxj.py` script, which parses each line with the `CDXObject` class from `pywb.warcserver.index.cdxobject`. For each record, it checks whether the MIME type contains "html" and the HTTP status code equals 200, then prints the corresponding URL. The visualization shows which URLs meet these criteria, highlighting successful HTML captures from the crawl data.

NN0taN3rd

58% match

Loading thumbnail…

Mat Kelly, PhD Student

This visualization examines the structure and contents of two WARC files from different crawls of Mat Kelly's academic homepage. It animates a series of frames that map the sequence of HTTP requests and responses, using a force-directed layout via `d3.forceSimulation` to position nodes by their relationships. The rendering combines SVG, Canvas, and WebGL to display the animation, with timing controlled by `transition.attrTween`. The data is parsed directly from the provided `heritrix.warc` and `wget.warc` files, visualizing crawl behavior and server responses like redirects and robots.txt blocks.

NN0taN3rd

56% match

Loading thumbnail…

WGET example

The visualization shows a network of hyperlinks and page relationships from a website crawl, using a force-directed graph layout. It employs D3.js’s `d3.forceSimulation` to position nodes (representing pages and resources) and `d3.link` to draw edges, with node size reflecting in-degree. The example is generated from a Wget command (`wget -mkEpnp http://example.org`) that mirrors a site, and the code parses the resulting HTML to extract links and build the graph. Data source is the locally mirrored site structure, rendered via SVG with transparent node fills and gray links, emphasizing connectivity patterns without animation or interaction.

HHugoberry

53% match

Loading thumbnail…

Gist 2d272134b57b02eb8cdd0c5904249591

This example tests the archival fidelity of a web page by loading a series of images and scripts that mimic dynamic behavior, including DOM manipulation, AJAX requests, and delayed updates. It uses JavaScript’s `XMLHttpRequest` to fetch remote images and replaces them via base64 data URLs, while `setTimeout` and scroll events trigger asynchronous content changes. The page is rendered within an iframe by the pywb web archiving tool, with a banner and rewrite layer applied. The visualization demonstrates how archived sites handle live interactions and resource loading over time.

NN0taN3rd

52% match

Loading thumbnail…

Gist f72a3553521c158f0eb0dccbe774455e

This example demonstrates a web scraper that extracts detailed trail information from a hiking website. It shows how structured data can be programmatically retrieved from HTML pages, specifically pulling out features, overview, and description sections. The code uses the Nokogiri library to parse the document and XPath queries to locate the relevant heading elements. After extracting each section, it cleans the text by removing newlines and excess whitespace. The scraper returns a hash containing the three descriptive fields, making the data readily accessible for further processing or visualization.

WWill Carter

51% match

Loading thumbnail…

Fetching data on web

This example demonstrates how to fetch and parse CSV data from a remote URL in a web browser. The code uses the `fetch` API to retrieve a plain-text file containing CSS named colors from a GitHub gist, then passes the response text to D3’s `csvParse` function for parsing. The parsed data is logged to the console via `console.log`. The example runs entirely in an HTML page with D3 v5 loaded from a CDN, without any additional framework or rendering to the screen.

NNita

51% match