I

IanHopkinson

@IanHopkinson·18 public vizzes

Loading thumbnail…

A gist illustrating testing in Python using unittest

This example demonstrates how to structure and execute a Python test suite using the built-in `unittest` framework, alongside `doctest` for embedded documentation checks. It shows the definition of a `factorial` function and a `TestFactorial` class that validates expected outputs, error handling, and edge cases. The code dynamically generates test methods via `setattr` and uses `subTest` for granular failure reporting. It also illustrates skipping tests with `@unittest.skip`, suppressing console logging via `logging.NullHandler`, and controlling output with `unittest.main(buffer=True)`. The visible data source is the hardcoded test case list and doctest examples within the function docstring.

Loading thumbnail…

csv concatenation

This example demonstrates the simple concatenation of multiple CSV files into a single output file, combining rows from all input files without any transformation or filtering. The script uses the `csv.reader` and `csv.writer` APIs from Python’s standard library to parse and write each row, while the `glob` module collects every `.csv` filename in a specified directory, and `os.chdir` changes the working directory to that folder. All rows are appended into a flat list, then written to a designated output path. The visualization shows the raw data-merging process, not a graphical chart, and relies solely on file I/O and CSV handling.

Loading thumbnail…

Modified start.sh for Docker Toolbox.

This example shows a Bash script that automates the setup of a Docker Toolbox virtual machine, specifically creating and starting the default VM and regenerating its certificates when needed. It demonstrates shell scripting for environment configuration, using commands like `docker-machine create`, `docker-machine start`, and `docker-machine env`, alongside VirtualBox’s `VBoxManage` to check VM existence. The script also defines a `docker` function that wraps `docker.exe` with `MSYS_NO_PATHCONV` to handle path conversion on Windows, and it prints a status message with the VM’s IP address. The code relies on standard Bash utilities and conditional logic, not on any external visualization library.

Loading thumbnail…

Third title

This example demonstrates how to use XPath queries with lxml to extract data from both HTML and XML documents. It shows techniques for selecting elements by tag name, attribute values, and text content, as well as navigating the document tree with axes like `following-sibling` and the `..` operator. The code uses `lxml.html.fromstring` and `lxml.etree.fromstring` to parse content fetched from a live website and a hardcoded XML sample, respectively. It highlights namespace handling in XML, showing that prefixed queries require explicit namespace mapping, while unprefixed queries fail without a default namespace binding. The example also illustrates retrieving attribute values like `setCount` and iterating over all elements with `getiterator`.

Loading thumbnail…

fresh block

This example demonstrates a minimal D3.js setup that initializes an SVG canvas and prepares a blank visualization area. It shows how to establish a chart’s margin convention and dimensions (960×500 pixels) while logging a confirmation message to the console. The code uses `d3.select` to append an SVG element, applies width and height attributes, and wraps it in a group with a translated transform. No data is loaded or rendered; the visualization merely lays the groundwork for future drawing, highlighting the basic scaffolding required before adding scales, axes, or marks.

Loading thumbnail…

Get people appearing with a list of people from NewsReader

This visualization maps the co-occurrence of people mentioned in news articles, showing which individuals appear alongside a list of queried names from NewsReader. The code retrieves data by iterating through paginated JSON requests to the NewsReader API, using the `requests` library to fetch actor pairs and their associated event counts. It filters results for names like David Beckham and Sepp Blatter, saving each co-occurrence record via `scraperwiki.sqlite` into an "actors" table. The example demonstrates a network-style data pipeline where connections between people are extracted from structured news metadata, though it does not include any client-side rendering or charting library.

Loading thumbnail…

Get a timeline from the NewsReader API

This visualization builds a chronological timeline of events involving the actor Mohammed bin Hammam by querying the NewsReader API, which returns event summaries linked to a specific DBpedia URI. It shows how each event’s size and timestamp are recorded, with the timeline displaying event occurrences over time. The code uses the `requests` library to fetch paginated JSON data, parsing the response to extract event details, and saves the results to an SQLite database via `scraperwiki.sqlite.save`. The timeline itself is not rendered here; the script only collects and stores the temporal data.

Loading thumbnail…

Get Yahoo!Finance data on ScraperWiki

This example shows how to pull historical stock price data from Yahoo! Finance and store it in a SQLite database. The code uses pandas’ `DataReader` with the `yahoo` API to fetch daily trading data for a given ticker symbol over a specified date range. It then resets the DataFrame index, converts the date column to ISO format, and writes the result to a table using `pandas.io.sql.write_frame` with a SQLite connection. The script demonstrates a straightforward data pipeline from an online financial API to local relational storage, using only the pandas and sqlite3 libraries.

Loading thumbnail…

Here are some things you can do with Gists in GistBox.

This example demonstrates how GistBox uses Gists to store and organize reusable code, functions, documentation, and even recipes. It shows the versatility of Gists across multiple languages, including JavaScript and Ruby, as well as Markdown for formatting guides. The code highlights specific APIs like `console.log` for browser interaction, a custom QuickSort implementation in Ruby, and embedded images and lists in Markdown. The visualization approach is textual, relying on syntax highlighting and file structure to convey organization, with the source being user-generated content from GitHub Gists.

Loading thumbnail…

R used to generate views of ScraperWiki's Twitter Search tool

This example visualizes the frequency of tweets over time, showing the number of tweets per minute from a ScraperWiki Twitter search dataset. The chart displays a histogram of tweet timestamps, with the x-axis representing dates and the y-axis showing tweet counts, capped at 40,000 tweets. The R script uses the `ggplot2` and `scales` libraries to bin the timestamps into 60-second intervals and render the plot. It queries data via ScraperWiki’s SQL interface, converts the `created_at` strings to POSIX date objects, and uses `qplot` and `ggplot_build` to construct and refine the histogram. The visualization highlights temporal activity patterns in the collected tweets.

Loading thumbnail…

Some R to knit a view on ScraperWiki

This example demonstrates how to knit an R view of ScraperWiki data into a static HTML report. The script uses the `knitr` library to render an Rhtml file, processing embedded R code and visualizations into a final web page. It relies on `render_html()` to generate the output and writes the result to an index.html file, making the data view portable and shareable. The code highlights a workflow for automating data-report generation from a ScraperWiki source.

Loading thumbnail…

A little bit of JavaScript that helps setup a box on ScraperWiki for tools inside that box

This example shows how to automate the setup of a tool environment on ScraperWiki by writing a dataset URL to a file and launching an installation script. The code uses ScraperWiki’s `scraperwiki.exec` and `scraperwiki.readSettings` APIs to run shell commands, while jQuery’s `$(function(){})` triggers the two setup functions on page load. It writes the target URL into `~/tool/dataset_url.txt` and runs `run-one tool/runonce.R` with logging to `tool/log.txt`. No data source or rendering is involved; the focus is purely on command-line orchestration within ScraperWiki’s execution context.

Loading thumbnail…

Gist 6112837

This example visualizes the 2008 cohort of vehicle test records from a gist, mapping individual vehicles as nodes in a force-directed network. The visualization uses d3.forceSimulation to compute node positions based on shared attributes like make and fuel type, while d3.geoPath renders a geographic backdrop for postcode regions. Canvas getImageData samples pixel colors for a density heatmap overlay, and transition.attrTween animates node movements during filtering. The schema.ini file defines the pipe-delimited data structure, and the code reads the CSV directly from the gist to drive the interactive layout.

Loading thumbnail…

SQL to upload the two big MOT data files to a MySQL database, requires the tables to by using the MyIASM engine.

This example demonstrates a SQL script that loads two large MOT test data files into a MySQL database. It first disables keys on the `testitem` and `testresult` tables, then uses `LOAD DATA LOCAL INFILE` to bulk-import tab-delimited text files, and finally re-enables keys to speed up the insertion. The code relies on MySQL’s `ALTER TABLE ... DISABLE KEYS` and `ENABLE KEYS` directives to optimize index maintenance for the MyISAM storage engine. The data source consists of two plain-text files, `UAT_test_item_2011.txt` and `UAT_test_result_2011.txt`, with fields separated by pipes, and the path is specified by the user.

Loading thumbnail…

SQL to upload the smaller tables of the MOT test dataset

This example demonstrates how to prepare smaller reference tables from the MOT test dataset for database loading, showing the SQL commands that import three text files—failure_location, item_detail, and item_group—into their corresponding MySQL tables. The script uses the `LOAD DATA LOCAL INFILE` API to parse pipe-delimited files with newline-terminated records, explicitly specifying field and line terminators. The code relies on standard SQL syntax within a MySQL environment, first selecting the MOT database with `USE MOT` before executing each import. This approach highlights a straightforward, file-based data ingestion pattern rather than a visual rendering technique, focusing on the database population step that precedes any charting or visualization work.

Loading thumbnail…

A schema for uploading MOT test data, as suggested in the user guide with the exception of the `SET storage_engine=MyISAM` which is there to allow the `ENABLE KEYS\DISABLE KEYS` mechanism used in the large table uploads. Also the field `RFRINSPMANDESC TEXT(500) #Variation from suggested schema due to 1074 error` is suggested as `char` in the user guide but for my MySQL setup this exceeds the allowed size for char.

This schema defines the relational structure for importing UK MOT test data, showing the primary tables for test results, test items, item details, item groups, and failure locations. It demonstrates the use of MySQL DDL with explicit primary keys, composite indexes, and the MyISAM storage engine to enable bulk-loading optimizations. The code highlights practical deviations from the official user guide, such as using `TEXT(500)` for inspection descriptions instead of an oversized `CHAR`, accommodating MySQL’s row-size limits while preserving data integrity.

Loading thumbnail…

A demonstration of doctests

This example demonstrates how to verify a function that filters histogram data using Python's `collections.Counter`. It shows a `threshold_above` function returning keys whose values exceed a given threshold, as confirmed by a doctest embedded in the docstring. The code imports `collections` and defines the function with a doctest example, illustrating the expected output for a sample counter. The visualization relies on Python’s standard library, specifically `collections.Counter` and doctest, to validate behavior without external plotting libraries. The example highlights testing practices for data-processing logic, not graphical rendering.

Loading thumbnail…

A gist in R showing how to plot twitter data from ScraperWiki as a smoothScatter plot

This example shows the relationship between follower and following counts for a subset of Twitter users, revealing a dense clustering pattern near the origin with sparse outliers at higher values. The R code first retrieves JSON data from a ScraperWiki SQL endpoint, converts it into a data frame using the `rjson` package, and then renders a smoothScatter plot—a base R function that uses hexagonal binning and color density to visualize overlapping points. The plot is restricted to a 0–5000 range on both axes to focus on the majority of users while still exposing the distribution’s shape. The approach highlights typical user behavior rather than individual outliers.