Skip to main content
100%

Edits to Rebecca Koeser's script to generate dataset from DH Q&A archive (preliminary)

✓ Published0🌍 Public
ZZoeLeBlanc
Last edited May 3, 2020
Created on May 3, 2020

This example shows a Python script that parses the Digital Humanities Q&A archive into a structured dataset, extracting questions, responses, authors, tags, dates, and best-answer flags from static HTML pages and linked RSS feeds. The script uses BeautifulSoup to locate page elements and pandas to organize post metadata into merged DataFrames, while the archive's source repository provides the raw HTML files. It also demonstrates a data-cleaning step that removes social-media widgets from post content before saving the final dataset.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist 7083cd8264b6aa48ce4bbfaf44546f07

This example builds a profile-scraping pipeline for social media data, turning raw CSV input into a structured DataFrame with fields for user names, follower counts, post dates, likes, comments, captions, and industry tags. The code reads `raw_data.csv` via `pandas.read_csv` and then initializes an empty DataFrame with predefined column names, preparing it for subsequent population. The visualization focuses on the data-preparation stage rather than graphical output, demonstrating how pandas handles tabular ingestion and schema definition. No rendering library like d3 or Three.js appears here; instead, the emphasis is on clean data structuring for later analysis or charting.

AArizonaTay
53% match
Loading thumbnail…

Gist 58c79c81e9a950a0cdafc7d2d0275742

This example shows a dynamic, browser-automated web-scraping workflow that captures rendered page content. It uses Selenium’s `webdriver` to launch a Chrome browser, wait for elements via `WebDriverWait` and `expected_conditions`, and then extracts page HTML with BeautifulSoup. The script programmatically sends keyboard inputs via `Keys`, and also employs the `requests` library for direct HTTP calls. It parses the returned data with `json`, timestamps actions with `datetime`, and structures results into a pandas DataFrame. The process includes randomized delays (`random` and `time`) to mimic human browsing behavior, though no explicit visualization or rendering library is used beyond the raw captured markup.

AArizonaTay
53% match
Loading thumbnail…

Social Media MQP Data Exploration

This example explores survey response data from a social media study, showing how answers are grouped by platform and question tag. The visualization displays the raw parsed data as a JSON string, revealing the frequency counts of each scaled response (like “Strongly Agree” or “Very Frequently”) per question. It uses d3.csvParse and d3.group to organize the data, with d3’s selection and join methods to render the text. The code fetches a remote CSV and processes the response counts client-side.

KK Munz
52% match
Loading thumbnail…

Gist dfbc4b41961e5b098fffce6af3828f57

This visualization scrapes Instagram profile data to compare follower counts, post engagement, and captions across influencers in different industries. Using Selenium to automate Chrome and BeautifulSoup to parse page JSON, the script extracts post dates, like counts, comments, and partnership flags from each profile’s embedded `_sharedData` blob. It then compiles the results into a pandas DataFrame and exports them to CSV. The example highlights the workflow of collecting social-media metrics at scale, with the final chart likely plotting follower growth or engagement trends per industry.

AArizonaTay
52% match
Loading thumbnail…

Gist fa05e9e0684e37e537c08085e5900435

This example shows a data-cleaning pipeline for an Instagram influencer dataset, highlighting the number of unique users and missing values across key columns. The code uses pandas to inspect and transform the data, filling missing industry labels with "General," dropping rows without captions, removing duplicate entries, and converting the post date column to datetime format for time-series indexing. The visualization relies on pandas' built-in data manipulation functions, such as `fillna`, `drop_duplicates`, and `set_index`, rather than a plotting library. The final output would be a cleaned dataframe ready for trend analysis, though no chart is generated in the provided script.

AArizonaTay
52% match
Loading thumbnail…

Social Media Analysis Loading Data

This example demonstrates the initial data-loading stage of a social media analysis dashboard. It fetches a CSV file from GitHub, filters out responses from participants who did not consent or are under 18, and restructures the raw survey columns into consolidated fields using a custom `combine` function. The raw data is displayed as JSON in a `<pre>` element. The process uses the `fetch` API, `csvParse`, and `select` from D3 v7, with state management handled through a custom `setState` callback.

Fflanagancarlie@gmail.com
51% match
Loading thumbnail…

XQuery for Partitioning the Blue Mountain Project

This example demonstrates a pipeline for cleaning, partitioning, and exporting articles from the Blue Mountain Project’s TEI transcriptions into language-specific databases. It shows how XQuery 3.1 scripts transform raw periodical XML by removing facsimile and hyphenation artifacts, splitting documents into individual articles, and grouping them by `mainLang` attributes into new databases. The workflow then exports French-language texts as plain .txt files, and R code using the `tm` and `SnowballC` libraries performs text mining, including term-document matrices and stemming.

CCliffordAnderson
51% match
Loading thumbnail…

download-hf-dataset-as-jsonl.ipynb

This example demonstrates an ETL pipeline that downloads a HuggingFace dataset of Wikipedia stubs about women in religion and converts it into JSONL files formatted for OpenAI fine-tuning. It shows how to transform raw dataset splits into the chat-based structure with system, user, and assistant roles. The notebook uses the `datasets` library’s `load_dataset` and `to_json` methods to handle training and validation splits separately, saving the output to `/content/`. A custom `create_openai_format` function then wraps each example in the conversational structure required by OpenAI’s API.

CCliffordAnderson
51% match