streaming-crawler.js that reads articles from first X pages

100%

streaming-crawler.js that reads articles from first X pages

✓ Published0🌍 Public

Last edited Nov 22, 2014

Created on Nov 22, 2014

This example streams the first five pages of Hacker News articles and extracts each story’s title and link. It uses the `request` library to fetch each page, then pipes the HTML through `html-tokenize` and `html-select` to parse elements matching the `.title a` selector. As text and attribute data flow through, the code logs each article’s href and title to the console, skipping links marked with `rel="nofollow"`. The process chains pages sequentially, incrementing the page number until reaching the stop condition, and finally reports the total number of pages crawled.

AI-generated description

Similar vizzes

Loading thumbnail…

request-hn.js

This example scrapes the front page of Hacker News and prints a plain-text list of story titles and their links to the console. It uses the `request` library to fetch the HTML and `cheerio` to parse the DOM, selecting elements with the `.title a` selector. The data is collected into an array of objects with `text` and `href` properties, then logged via `console.log`. No visualization is rendered; the output is purely textual.

Aalessioalex

72% match

Loading thumbnail…

A simple news crawler.

This example demonstrates a news crawler that fetches headlines from four Malaysian RSS feeds, extracts the full article text, and stores each story as a JSON file organized by publication date. The crawler uses `feedparser` to parse RSS feeds, `requests` to download article pages, and the `readability` library to isolate main content from HTML. `BeautifulSoup` cleans the extracted text, while `dateutil.parser` converts publication timestamps into structured day, month, and year fields. Each JSON file, saved in a local `collection` folder, contains the article’s cleaned content, original link, and parsed date components.

Aamirothman

62% match

Loading thumbnail…

2017-07-25: Swimming In A Ocean Of JavaScript. Using Nodejs, Electron And Chrome To Preserve The Web

This example demonstrates a Node.js toolkit for web archiving, showing how to parse, stream, and generate WARC and CDXJ files to preserve web content. It uses the `cdxj` library to read CDXJ indexes asynchronously, the `node-warc` parser with event listeners for streaming WARC records, and a configuration object for the `squid` crawler that controls Chrome via its WebSocket API. The code highlights handling HTTP/2 limitations in archives and supports features like page-same-domain crawling and auto-scrolling.

NN0taN3rd

58% match

Loading thumbnail…

Gist f72a3553521c158f0eb0dccbe774455e

This example demonstrates a web scraper that extracts detailed trail information from a hiking website. It shows how structured data can be programmatically retrieved from HTML pages, specifically pulling out features, overview, and description sections. The code uses the Nokogiri library to parse the document and XPath queries to locate the relevant heading elements. After extracting each section, it cleans the text by removing newlines and excess whitespace. The scraper returns a hash containing the three descriptive fields, making the data readily accessible for further processing or visualization.

WWill Carter

56% match

Loading thumbnail…

use chrome to crawl

This example demonstrates automated web crawling and interaction using Chrome's remote interface. The code navigates through URLs from a seed list, captures page HTML and window environment variables, and programmatically clicks all elements with click event listeners. It manages JavaScript dialogs and navigation timeouts, while logging errors. It relies on Chrome's Remote Interface (CDP), along with Bluebird for promise handling, and custom detectors for JS library identification.

NN0taN3rd

56% match

Loading thumbnail…

Gist 5ea363dfe112d08281a8c89a3f31bcff

This example scrapes trail data from a website, extracting features, overview, and description sections into a structured hash. It uses Nokogiri to parse the HTML and XPath queries to locate the relevant headings and their adjacent content, cleaning the text by removing line breaks. The code demonstrates a straightforward scraping pattern for capturing semantic sections, though it does not include visualization logic or rendering libraries. The data source is a single trail page URL, and the output is a formatted description ready for further processing or display.

WWill Carter

55% match

Loading thumbnail…

Instacart Scraper

This example shows a browser extension that automatically visits a queue of URLs and captures matching JSON responses from XHR and fetch requests. It displays real-time progress through the URL queue, tracking completed and pending items, active page loads, and downloaded file counts. The visualization updates every second using popup.js polling with chrome.runtime.sendMessage, while injected.js intercepts network calls before they reach the page and content.js forwards captured data to the background service worker, which saves each response as a JSON file.

SStephen

55% match

Loading thumbnail…

Press release scrape for pressreleasepoint.com - verbose output to console and saves to text file

This example scrapes press releases from pressreleasepoint.com, saving each article’s text to individual files while printing both the download URL and content to the console. It demonstrates web scraping with BeautifulSoup and requests, iterating through 2,497 paginated index pages and extracting text from the `content-press_release` class. The script handles missing links with a broad exception pass, outputs verbose progress logs, and organizes saved files by page and index.

Aabehmiel

55% match