How to use The Guardian's API to download article data for content analysis (in Python 3.x)

100%

How to use The Guardian's API to download article data for content analysis (in Python 3.x)

✓ Published0🌍 Public

Last edited Mar 7, 2016

Created on Mar 8, 2016

This example demonstrates how to perform content analysis on Guardian news articles by programmatically downloading article data through The Guardian’s open API. It shows the entire workflow, from constructing API queries with parameters like `from-date`, `to-date`, and `show-fields`, to parsing the returned JSON metadata and article bodies. The Python script uses the `requests` library to fetch paginated results into day-sized chunks, saving the raw JSON files locally for further analysis. The example also highlights the API’s response structure, including fields such as `wordcount`, `byline`, and full article HTML.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist e71cb443307994e24e63465bac636d3e

This example demonstrates how to extract and summarize the content of a New York Times opinion article about regulating artificial intelligence. The notebook uses the `newspaper` library to download and parse the article, returning its title, publication date, and initial text. It then calls the `nlp()` method to generate a list of keywords and a multi-sentence summary, showing a straightforward pipeline from a URL to condensed article insights. The code relies on `Article` from `newspaper` and standard pandas imports.

GGeorgeMcIntire

61% match

Loading thumbnail…

Gist d3be7947ba494f0cc785ad4ff966f6d5

This example demonstrates a workflow for collecting and structuring Twitter data through the streaming API. It uses the `tweepy` library to authenticate and open a live stream, filtering tweets by specified terms or hashtags, and saving each raw tweet as a JSON line to a file. The code then reads that file back, parses each tweet, and extracts selected fields into a nested list, including the text, user handle, favorite and retweet counts, timestamp, follower count, and language. Finally, it organizes the data into a structured pandas DataFrame with corresponding column names, preparing it for further analysis or visualization.

GGeorgeMcIntire

60% match

Loading thumbnail…

Gist d003ece0b6ca171be8e095e0bc90af7e

This notebook example demonstrates how to programmatically scrape, parse, and structure a corpus of New York Times articles into a pandas DataFrame. The code loads a pre-saved pickle file of 50 article links, iterates through each URL using the `newspaper` library's `Article` class, and extracts the article’s text, title, publication date, keywords, and summary. It then stores the results in a list of dictionaries and converts that list into a structured DataFrame. The script includes error handling for unparseable articles, inserting `NaN` values, and prints the total execution time, which took roughly 38.5 seconds for this run. The displayed table shows the resulting columns: date, keywords, link, summary, text, and title.

GGeorgeMcIntire

60% match

Loading thumbnail…

A vanilla py-script to download all Pocket.com data via API.

This example demonstrates how to download a complete archive of all Pocket.com saved articles using the Pocket API. It shows the OAuth authentication flow, which opens a browser for user approval, then retrieves every article in batches of ten, displaying each article’s title and URL as it processes them. The script uses Python’s `requests` library to make HTTP POST calls to the Pocket API endpoints, along with the `json` module to parse responses and `subprocess` to trigger browser opening. It saves the full article list as a JSON file.

BBastiTee

59% match

Loading thumbnail…

Gist d2f2bc6be1716820160ed5183ea07eaa

This example shows a browser-based text analytics pipeline that scrapes and processes web content before visualizing topic models. The code first uses Selenium with Chrome WebDriver to scrape data, then applies NLP libraries including NLTK, spaCy, TextBlob, and gensim’s LdaModel for topic modeling and coherence scoring. The final visualization employs Plotly’s `plotly.express` and `plotly.graph_objs` to render interactive charts, with `pyo.init_notebook_mode()` enabling notebook display. The rendering approach is evident from the use of `go` traces and `px` figures, though the specific chart type is not shown in the code snippet. The data source is dynamic web pages, as indicated by the Selenium scraping logic.

AArizonaTay

53% match

Loading thumbnail…

Gist 8044140b08e7e7887e0c7531911d1fa4

This example streams live tweets matching hashtags like #DoctorWho and #DrWho, saving each tweet’s metadata to a CSV file using the Twitter API and the `twitter` Python package. It demonstrates real-time data collection, extracting fields such as user location, follower count, and coordinates. The code handles truncated tweets by fetching extended text, and it cleans nested dictionary values with helper functions. No visualization is rendered; the focus is purely on acquiring and structuring streaming social media data for later analysis.

AAjar06

53% match

Loading thumbnail…

Gist 0fbf0f6cbe67ab63e90d47d65cda05ea

This visualization demonstrates a three-stage search pipeline that locates content across numerous blog-post dialogs without loading entire files. Using Python's async/await syntax, the code calls three functions—`get_first_notes`, `get_section_summaries`, and `get_section_content`—in bulk loops to filter a folder down to relevant sections, reducing token usage from tens of thousands to roughly six thousand. The transitions between stages are driven by agent analysis of summaries, with each stage executing as a single kernel call.

深深度碎片

52% match

Loading thumbnail…

Gist 58c79c81e9a950a0cdafc7d2d0275742

This example shows a dynamic, browser-automated web-scraping workflow that captures rendered page content. It uses Selenium’s `webdriver` to launch a Chrome browser, wait for elements via `WebDriverWait` and `expected_conditions`, and then extracts page HTML with BeautifulSoup. The script programmatically sends keyboard inputs via `Keys`, and also employs the `requests` library for direct HTTP calls. It parses the returned data with `json`, timestamps actions with `datetime`, and structures results into a pandas DataFrame. The process includes randomized delays (`random` and `time`) to mimic human browsing behavior, though no explicit visualization or rendering library is used beyond the raw captured markup.

AArizonaTay

52% match