Gist 2559ad6fd17831c9259a8af6f38ec027

100%

Gist 2559ad6fd17831c9259a8af6f38ec027

✓ Published0🌍 Public

Last edited Jun 14, 2018

Created on Jun 14, 2018

This example filters a Common Crawl index file to identify archived web pages. It reads a CDXJ file using the `read_cdxj.py` script, which parses each line with the `CDXObject` class from `pywb.warcserver.index.cdxobject`. For each record, it checks whether the MIME type contains "html" and the HTTP status code equals 200, then prints the corresponding URL. The visualization shows which URLs meet these criteria, highlighting successful HTML captures from the crawl data.

AI-generated description

Similar vizzes

Loading thumbnail…

CS 891 Web Archiving Seminar

This visualization demonstrates the process of web archiving by showing the WARC file records produced when using `wget` to archive two websites: `http://example.com` and the React Router documentation. The code uses `wget` with extensive arguments including `--mirror`, `--warc-cdx`, and `--page-requisites` to capture complete page snapshots. The example displays the resulting WARC file structure, containing request, response, metadata, and resource records, along with a shell script that orchestrates the archiving workflow and manages the Wayback Machine collections.

NN0taN3rd

59% match

Loading thumbnail…

Dump warc

This example shows a tool that extracts and saves web resources from a WARC archive file into a local directory, organizing files by their original URLs and content types. It parses the archive using the `node-warc` library in JavaScript or `warcio` in Python, mapping HTTP GET requests to their corresponding 200 OK responses. The code uses `filenamify-url` or `goldfinch` to sanitize filenames, and `fs-extra` or `pathlib` for file operations, writing each HTML, image, CSS, or JavaScript file with an appropriate file extension based on the response’s Content-Type header.

NN0taN3rd

58% match

Loading thumbnail…

Gist 2d272134b57b02eb8cdd0c5904249591

This example tests the archival fidelity of a web page by loading a series of images and scripts that mimic dynamic behavior, including DOM manipulation, AJAX requests, and delayed updates. It uses JavaScript’s `XMLHttpRequest` to fetch remote images and replaces them via base64 data URLs, while `setTimeout` and scroll events trigger asynchronous content changes. The page is rendered within an iframe by the pywb web archiving tool, with a banner and rewrite layer applied. The visualization demonstrates how archived sites handle live interactions and resource loading over time.

NN0taN3rd

58% match

Loading thumbnail…

2017-07-25: Swimming In A Ocean Of JavaScript. Using Nodejs, Electron And Chrome To Preserve The Web

This example demonstrates a Node.js toolkit for web archiving, showing how to parse, stream, and generate WARC and CDXJ files to preserve web content. It uses the `cdxj` library to read CDXJ indexes asynchronously, the `node-warc` parser with event listeners for streaming WARC records, and a configuration object for the `squid` crawler that controls Chrome via its WebSocket API. The code highlights handling HTTP/2 limitations in archives and supports features like page-same-domain crawling and auto-scrolling.

NN0taN3rd

58% match

Loading thumbnail…

WGET example

The visualization shows a network of hyperlinks and page relationships from a website crawl, using a force-directed graph layout. It employs D3.js’s `d3.forceSimulation` to position nodes (representing pages and resources) and `d3.link` to draw edges, with node size reflecting in-degree. The example is generated from a Wget command (`wget -mkEpnp http://example.org`) that mirrors a site, and the code parses the resulting HTML to extract links and build the graph. Data source is the locally mirrored site structure, rendered via SVG with transparent node fills and gray links, emphasizing connectivity patterns without animation or interaction.

HHugoberry

56% match

Loading thumbnail…

Gist 41e56c91ab9d245a846939ab9a448e40

This visualization tracks the archival status of web resources across years, showing counts of collected, archived, and not-archived items. It uses a grouped bar chart or line chart to compare yearly totals, with stacked segments for different archive services like archive.org, archive-it.org, and archive.today. The code employs D3.js’s `d3.forceSimulation` for layout and `transition.attrTween` for animated updates, while rendering via SVG paths. Data comes from the provided CSV file, which is parsed using `d3.csv` and mapped to scales for year and count.

HHussamHallak

56% match

Loading thumbnail…

Gist 18dc25595ce9bc0bf23c259cb0829993

This visualization tracks how often archived web pages from the Internet Archive were collected each year between 1999 and 2022, using a stacked bar chart built with D3.js. It shows both the total number of archived items per year and how many were successfully collected versus those that were not archived, broken down by archival sources like archive.org, archive-it.org, and archive.today. The chart uses `d3.forceSimulation` for layout adjustments and `d3.geoPath` for any geographic elements, while rendering bars with SVG paths. Data comes from a CSV file containing yearly counts of archived and unarchived records, with color encoding to distinguish between archive providers and error cases.

HHussamHallak

55% match

Loading thumbnail…

Gist d2f2bc6be1716820160ed5183ea07eaa

This example shows a browser-based text analytics pipeline that scrapes and processes web content before visualizing topic models. The code first uses Selenium with Chrome WebDriver to scrape data, then applies NLP libraries including NLTK, spaCy, TextBlob, and gensim’s LdaModel for topic modeling and coherence scoring. The final visualization employs Plotly’s `plotly.express` and `plotly.graph_objs` to render interactive charts, with `pyo.init_notebook_mode()` enabling notebook display. The rendering approach is evident from the use of `go` traces and `px` figures, though the specific chart type is not shown in the code snippet. The data source is dynamic web pages, as indicated by the Selenium scraping logic.

AArizonaTay

55% match