Skip to main content
100%

Buzzfeed article scraper for NLP

✓ Published0🌍 Public
Aabehmiel
Last edited Oct 24, 2017
Created on Oct 6, 2017

This example scrapes a Buzzfeed article and converts its text into clean, tokenized sentences for natural language processing. It shows the pipeline from raw HTML to normalized, sentence-segmented data. The scraper uses BeautifulSoup and requests to extract paragraph elements with the class `subbuzz-text`. Text cleaning relies on the `unicodedata` module and regex via `re`, while sentence tokenization uses NLTK’s Punkt tokenizer. The final output is a single string of joined sentences, ready for further NLP analysis.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist d2f2bc6be1716820160ed5183ea07eaa

This example shows a browser-based text analytics pipeline that scrapes and processes web content before visualizing topic models. The code first uses Selenium with Chrome WebDriver to scrape data, then applies NLP libraries including NLTK, spaCy, TextBlob, and gensim’s LdaModel for topic modeling and coherence scoring. The final visualization employs Plotly’s `plotly.express` and `plotly.graph_objs` to render interactive charts, with `pyo.init_notebook_mode()` enabling notebook display. The rendering approach is evident from the use of `go` traces and `px` figures, though the specific chart type is not shown in the code snippet. The data source is dynamic web pages, as indicated by the Selenium scraping logic.

AArizonaTay
62% match
Loading thumbnail…

NLP Concepts with spaCy. Code examples released under CC0 https://creativecommons.org/choose/zero/, other text released under CC BY 4.0 https://creativecommons.org/licenses/by/4.0/

This notebook introduces core NLP concepts using the spaCy library, demonstrating how to analyze text structure through part-of-speech tagging and dependency parsing. It explains grammatical abstractions like sentences and word functions while acknowledging their limitations, particularly for poetry. The code example uses spaCy’s built-in pipeline to process English sentences, showing token attributes and linguistic annotations. Written in Python 2.7, it also mentions alternatives like NLTK and resources for other languages. The notebook’s content mixes explanatory prose with executable cells, offering a beginner-friendly tour of how language can be computationally decomposed.

Aabehmiel
60% match
Loading thumbnail…

A simple news crawler.

This example demonstrates a news crawler that fetches headlines from four Malaysian RSS feeds, extracts the full article text, and stores each story as a JSON file organized by publication date. The crawler uses `feedparser` to parse RSS feeds, `requests` to download article pages, and the `readability` library to isolate main content from HTML. `BeautifulSoup` cleans the extracted text, while `dateutil.parser` converts publication timestamps into structured day, month, and year fields. Each JSON file, saved in a local `collection` folder, contains the article’s cleaned content, original link, and parsed date components.

Aamirothman
59% match
Loading thumbnail…

Python Utilities for Tweets

This utility module processes raw tweet JSON objects for text analysis. It extracts timestamps via `datetime.strptime`, hashtags, user mentions, and cleaned text by stripping URLs, mentions, and media using entity indices. It further sanitizes punctuation with the `string` module, removes English stopwords from `nltk.corpus`, and applies Lancaster stemming via `nltk.stem.lancaster.LancasterStemmer`. The code demonstrates reusable functions for preparing Twitter data, relying solely on standard Python and NLTK libraries without any visualization or rendering components.

Aabehmiel
58% match
Loading thumbnail…

Gist e71cb443307994e24e63465bac636d3e

This example demonstrates how to extract and summarize the content of a New York Times opinion article about regulating artificial intelligence. The notebook uses the `newspaper` library to download and parse the article, returning its title, publication date, and initial text. It then calls the `nlp()` method to generate a list of keywords and a multi-sentence summary, showing a straightforward pipeline from a URL to condensed article insights. The code relies on `Article` from `newspaper` and standard pandas imports.

GGeorgeMcIntire
58% match
Loading thumbnail…

Press release scrape for pressreleasepoint.com - verbose output to console and saves to text file

This example scrapes press releases from pressreleasepoint.com, saving each article’s text to individual files while printing both the download URL and content to the console. It demonstrates web scraping with BeautifulSoup and requests, iterating through 2,497 paginated index pages and extracting text from the `content-press_release` class. The script handles missing links with a broad exception pass, outputs verbose progress logs, and organizes saved files by page and index.

Aabehmiel
56% match
Loading thumbnail…

Gist 55fc864ec9cea1107845d3ed7a3dab8b

This example demonstrates a text-cleaning pipeline for caption data, showing how raw captions are converted into tokenized, filtered lists for downstream analysis. The visualization itself is not rendered; instead, the focus is on the data preprocessing logic. The code uses `pandas` with `apply` to run natural language processing via `spaCy` (`nlp`), extracting tokens from the `Caption` column. It then applies a custom `tokens` function that removes stop words and non-alphanumeric characters using `re.match`, while preserving tokens containing "@" symbols. The result is a new `clean_caption` column, highlighting a common step in text mining workflows.

AArizonaTay
56% match
Loading thumbnail…

Gist d003ece0b6ca171be8e095e0bc90af7e

This notebook example demonstrates how to programmatically scrape, parse, and structure a corpus of New York Times articles into a pandas DataFrame. The code loads a pre-saved pickle file of 50 article links, iterates through each URL using the `newspaper` library's `Article` class, and extracts the article’s text, title, publication date, keywords, and summary. It then stores the results in a list of dictionaries and converts that list into a structured DataFrame. The script includes error handling for unparseable articles, inserting `NaN` values, and prints the total execution time, which took roughly 38.5 seconds for this run. The displayed table shows the resulting columns: date, keywords, link, summary, text, and title.

GGeorgeMcIntire
56% match