Skip to main content
100%

A simple news crawler.

✓ Published0🌍 Public
Aamirothman
Last edited May 25, 2016
Created on May 25, 2016

This example demonstrates a news crawler that fetches headlines from four Malaysian RSS feeds, extracts the full article text, and stores each story as a JSON file organized by publication date. The crawler uses `feedparser` to parse RSS feeds, `requests` to download article pages, and the `readability` library to isolate main content from HTML. `BeautifulSoup` cleans the extracted text, while `dateutil.parser` converts publication timestamps into structured day, month, and year fields. Each JSON file, saved in a local `collection` folder, contains the article’s cleaned content, original link, and parsed date components.

AI-generated description

Similar vizzes

Loading thumbnail…

streaming-crawler.js that reads articles from first X pages

This example streams the first five pages of Hacker News articles and extracts each story’s title and link. It uses the `request` library to fetch each page, then pipes the HTML through `html-tokenize` and `html-select` to parse elements matching the `.title a` selector. As text and attribute data flow through, the code logs each article’s href and title to the console, skipping links marked with `rel="nofollow"`. The process chains pages sequentially, incrementing the page number until reaching the stop condition, and finally reports the total number of pages crawled.

Aalessioalex
62% match
Loading thumbnail…

Buzzfeed article scraper for NLP

This example scrapes a Buzzfeed article and converts its text into clean, tokenized sentences for natural language processing. It shows the pipeline from raw HTML to normalized, sentence-segmented data. The scraper uses BeautifulSoup and requests to extract paragraph elements with the class `subbuzz-text`. Text cleaning relies on the `unicodedata` module and regex via `re`, while sentence tokenization uses NLTK’s Punkt tokenizer. The final output is a single string of joined sentences, ready for further NLP analysis.

Aabehmiel
59% match
Loading thumbnail…

Gist 5ea363dfe112d08281a8c89a3f31bcff

This example scrapes trail data from a website, extracting features, overview, and description sections into a structured hash. It uses Nokogiri to parse the HTML and XPath queries to locate the relevant headings and their adjacent content, cleaning the text by removing line breaks. The code demonstrates a straightforward scraping pattern for capturing semantic sections, though it does not include visualization logic or rendering libraries. The data source is a single trail page URL, and the output is a formatted description ready for further processing or display.

WWill Carter
56% match
Loading thumbnail…

Gist f72a3553521c158f0eb0dccbe774455e

This example demonstrates a web scraper that extracts detailed trail information from a hiking website. It shows how structured data can be programmatically retrieved from HTML pages, specifically pulling out features, overview, and description sections. The code uses the Nokogiri library to parse the document and XPath queries to locate the relevant heading elements. After extracting each section, it cleans the text by removing newlines and excess whitespace. The scraper returns a hash containing the three descriptive fields, making the data readily accessible for further processing or visualization.

WWill Carter
56% match
Loading thumbnail…

request-hn.js

This example scrapes the front page of Hacker News and prints a plain-text list of story titles and their links to the console. It uses the `request` library to fetch the HTML and `cheerio` to parse the DOM, selecting elements with the `.title a` selector. The data is collected into an array of objects with `text` and `href` properties, then logged via `console.log`. No visualization is rendered; the output is purely textual.

Aalessioalex
56% match
Loading thumbnail…

Press release scrape for pressreleasepoint.com - verbose output to console and saves to text file

This example scrapes press releases from pressreleasepoint.com, saving each article’s text to individual files while printing both the download URL and content to the console. It demonstrates web scraping with BeautifulSoup and requests, iterating through 2,497 paginated index pages and extracting text from the `content-press_release` class. The script handles missing links with a broad exception pass, outputs verbose progress logs, and organizes saved files by page and index.

Aabehmiel
56% match
Loading thumbnail…

Script to download all fox news press releases. Prints verbose output to console and file.

This example scrapes Fox News press releases from the press archive for years 2011–2017, organized by month, and saves each release as a separate text file. It uses BeautifulSoup and the `requests` library to parse HTML pages, extract article text from elements with class `hentry`, and prints verbose download progress and content to the console while writing files to a `press-releases/` directory. The script iterates through URL patterns for each year and month, handling missing pages silently.

Aabehmiel
56% match
Loading thumbnail…

Fork of HTML Starter

This example renders a static social media feed interface with HTML and CSS, mimicking the layout of a typical news feed. It displays several posts with author names, timestamps, and content, including a text-heavy post on Buddhist philosophy. The design uses a clean, minimalist style with a white header, a container for posts, and action links like “Like,” “Comment,” and “Share.” No external JavaScript libraries or data APIs are used; the feed is entirely hard-coded in the HTML document, with styling handled by embedded CSS rules for layout, spacing, and typography.

NNita
55% match