100%

Requires

✓ Published0🌍 Public

Last edited Sep 17, 2017

Created on Aug 20, 2017

This example demonstrates scraping public Twitter profiles with Puppeteer, a headless Chrome automation library, to extract tweet content, timestamps, embedded image URLs, and adaptive media containers. It also shows scraping a live webpage via the Wayback Machine to capture a full-page screenshot at 1920x1080 resolution. The script uses Puppeteer’s `page.evaluate` to run DOM queries, Bluebird’s `Promise.delay` for controlled waiting, and `fs-extra` to write the collected tweet data as JSON. It navigates pages, clicks "show more" links, and scrolls to trigger dynamic content loading before saving results.

AI-generated description

Requires

Node.js download

Usage

npm install
npm run go

Other Info

Comments in js file and have fun!

MIT Licensed

Similar vizzes

Loading thumbnail…

Gist 58c79c81e9a950a0cdafc7d2d0275742

This example shows a dynamic, browser-automated web-scraping workflow that captures rendered page content. It uses Selenium’s `webdriver` to launch a Chrome browser, wait for elements via `WebDriverWait` and `expected_conditions`, and then extracts page HTML with BeautifulSoup. The script programmatically sends keyboard inputs via `Keys`, and also employs the `requests` library for direct HTTP calls. It parses the returned data with `json`, timestamps actions with `datetime`, and structures results into a pandas DataFrame. The process includes randomized delays (`random` and `time`) to mimic human browsing behavior, though no explicit visualization or rendering library is used beyond the raw captured markup.

AArizonaTay

63% match

Loading thumbnail…

Usage

This example demonstrates how to replay a historical archived version of the CNN homepage through a headless Chrome browser. It loads a specific Wayback Machine capture from December 2017, setting the viewport to a fixed 1920×1080 resolution. The script uses Puppeteer to launch the browser, and injects a custom `magic` function that overrides `document.domain` to mimic the live site’s domain for proper page behavior. The navigation targets the archived URL directly, with a 60-second timeout, and any errors are logged to the console.

NN0taN3rd

62% match

Loading thumbnail…

use chrome to crawl

This example demonstrates automated web crawling and interaction using Chrome's remote interface. The code navigates through URLs from a seed list, captures page HTML and window environment variables, and programmatically clicks all elements with click event listeners. It manages JavaScript dialogs and navigation timeouts, while logging errors. It relies on Chrome's Remote Interface (CDP), along with Bluebird for promise handling, and custom detectors for JS library identification.

NN0taN3rd

61% match

Loading thumbnail…

Take a snapshot of a page with phantom JS

This example captures a static snapshot of a live web page using PhantomJS. It loads the DC Metro Metrics daily service report for February 23, 2015, then waits five seconds before writing the rendered HTML content to standard output, producing a full page dump. The code relies on PhantomJS’s `webpage` and `system` modules, setting a Chrome user-agent and using `page.open` with a callback and `setTimeout` to ensure the page finishes rendering. The result is a saved HTML file that preserves the report’s final visual state.

LLeeMendelowitz

60% match

Loading thumbnail…

Gist 2d272134b57b02eb8cdd0c5904249591

This example tests the archival fidelity of a web page by loading a series of images and scripts that mimic dynamic behavior, including DOM manipulation, AJAX requests, and delayed updates. It uses JavaScript’s `XMLHttpRequest` to fetch remote images and replaces them via base64 data URLs, while `setTimeout` and scroll events trigger asynchronous content changes. The page is rendered within an iframe by the pywb web archiving tool, with a banner and rewrite layer applied. The visualization demonstrates how archived sites handle live interactions and resource loading over time.

NN0taN3rd

56% match

Loading thumbnail…

systemjs-twitter-widgets

This example demonstrates how to load Twitter’s embedded-tweet widget through the SystemJS module loader. It shows a single tweet rendered via the platform.twitter.com widgets script, using SystemJS configuration to treat the Twitter library as a global script dependency. The code defines a custom `meta` entry for the `twitter` path, setting `scriptLoad: true` and `format: "global"` to bypass XHR restrictions, while `build: false` prevents bundling. The `index.js` file simply requires the Twitter module, triggering the widget to hydrate the blockquote in the HTML. The visualization highlights SystemJS’s flexibility with third-party, non-ES6 APIs.

HHypercubed

56% match

Loading thumbnail…

Gist a99d287be62c6d73536521c0e5e74e58

This example scrapes Instagram profile and post data to track follower counts, likes, comments, captions, and post dates for selected users, associating each entry with an industry topic. The script uses Selenium to automate login and search, BeautifulSoup to parse the page source, and the Instagram GraphQL data embedded in `window._sharedData` to extract metrics. It writes the collected records to a CSV via pandas, pausing with random sleeps to avoid detection. The visualization highlights how social-media engagement varies across profiles and industries over time.

AArizonaTay

56% match

Loading thumbnail…

html: load latest tweet

This example fetches and displays the user’s most recent tweet, showing the tweet text and associated metadata in a simple card layout. It uses the Fetch API to load data from a local or remote JSON endpoint, then renders the content into an unordered list with blockquotes and figure elements. The code relies on standard DOM manipulation methods to insert the tweet’s text and any embedded media or author details, though no external visualization libraries are employed. The layout is styled with minimal CSS, and the data source is assumed to be a JSON response containing the latest tweet object. The figure element is left empty, suggesting it may hold an image or avatar in a fuller implementation.

BBraMKJ

55% match