RobColeman

@RobColeman·10 public vizzes

Gist 1dcc8a9bd27317b89fdde2c22c049100

This example presents a Scala-based feature engineering pipeline for machine learning models, specifically for predicting eCPM (effective cost per mille) in advertising. The code defines a system of feature computers that transform prediction requests into sparse feature vectors using Apache Spark's MLlib library (`SparkVectors.sparse`) and the `MurmurHash3` hashing function. It shows how categorical features like app publisher and model identifiers are hashed into indexed blocks within a numeric vector, with each block's position and size tracked through `FeatureComputerParams`. The implementation uses pattern matching in `FeatureComputerRouter` to select appropriate computations and aggregates values into a map that can be converted to Spark's sparse vector format for model training.

Aug 24, 2016

Loading thumbnail…

An Approximate Distribution wrapper class for TDunnings java TDigest library

This example demonstrates how to estimate and visualize approximate probability distributions from large datasets using a wrapper class around TDunning's Java TDigest library. The code shows both local and Spark-based aggregation paths, where data is converted into TreeDigest instances and merged via reduction operations. It generates CDF and PDF sequences by sampling support points between observed minimum and maximum values, then plots these distributions for comparison. The example also illustrates Kryo serialization configuration for Spark clusters, using 100,000 samples drawn from a normal distribution to validate the approach.

Jan 29, 2016

Loading thumbnail…

Helpers for TDunnings Java TDigest library

This example demonstrates how to use the TreeDigest implementation from the Java TDigest library to compute approximate cumulative distribution functions (CDFs) and probability density functions (PDFs) from large datasets. It shows three Scala workflows: serializing and deserializing digests, merging per-element digests across an Apache Spark RDD using `reduce`, and using Kryo serialization to pass TreeDigest objects directly between Spark tasks. The code samples from exponential and normal distributions via Apache Commons Math, then compares CDF values computed locally versus through Spark, highlighting the library’s integration with distributed data processing.

Jan 28, 2016

Loading thumbnail…

A fast, serializable, Scala implementation of tDigest. Thrown together for a Spark project.

This visualization demonstrates a serializable Scala implementation of the tDigest algorithm, designed for fast approximate quantile estimation in Apache Spark. It defines `Centroid` and `TDigest` case classes that use Scala’s mutable `TreeSet` for sorted centroid storage, with methods for updating, merging, compressing, and computing cumulative distribution and inverse CDF values. The code relies on `org.apache.spark.SparkContext` and `org.apache.commons.math3.distribution.ExponentialDistribution`, using `scala.util.Random` for centroid shuffling during merges.

Jan 28, 2016

Loading thumbnail…

Some Algorithms In Python

This example illustrates a Python implementation of the binary search algorithm, demonstrating how it recursively bisects a sorted array to locate a target value. The code defines the `binary_search` function, which prints diagnostic messages at each split and returns a boolean result. It also includes a `run_binary_search` function that searches for the integer 6 in a range of 0 through 99. The visualization shows the algorithm's step-by-step process through console output rather than graphical rendering, using standard Python logic and print statements.

Nov 9, 2015

Loading thumbnail…

Counting nucleotides from a sequence

This example shows how to calculate the percentage of G and C nucleotides in a DNA sequence, a common metric in genomics. It first counts each nucleotide type (A, C, G, T) using a Python dictionary, then normalizes those counts by the total sequence length to compute percentages. The code defines two functions, `count_nucleotides_by_type` and `normalize_nucleotide_count`, and finally prints the combined G and C percentage for a hardcoded sequence string. No external visualization libraries are used; the output is a single text result.

Oct 27, 2015

Loading thumbnail…

Gist 2babd0fad3d7f01f3d7e

This example demonstrates how to count numeric digits within a string, with functions that either return the count or print it. The code uses Python’s built-in `str.isdigit()` method to check each character, iterating with a `for` loop and maintaining a counter. It tests the function on several sample strings, including those with no numbers, one number, and multiple digits, showing the output for each case. The second function, `how_many_numbers_in_the_word_print`, shows the same logic but uses `print` to display the result directly instead of returning it.

May 22, 2015

Loading thumbnail…

Loading CSV Data with D3

This example demonstrates how to load and inspect external CSV data using D3. It uses `d3.csv` to fetch the meteorites dataset and converts each row into a JavaScript object, which is then output to the browser’s console. The page itself displays only a brief note, while the actual data—including meteorite names, masses, and fall years—is visible through the developer tools. The code relies on D3 v3’s `d3.csv` method for file loading and parsing.

Apr 3, 2015

Loading thumbnail…

d3_course_dataset_meteorites

This visualization maps meteorite landings from the Meteorite Society dataset, showing each fall as a point on a geographic projection. It uses D3’s `d3.geoPath` to render a world map and `d3.forceSimulation` to position or jitter overlapping points. The color and size of the circles encode the meteorite’s class and mass, while the tooltip reveals details like year and name. The rendering relies on SVG paths and D3 scales, with the data drawn from the cleaned CSV file hosted at visualizing.org.

Apr 2, 2015

Loading thumbnail…

Gist 9bebbdbcb59d4945538e

This example demonstrates how to estimate and visualize neural connectivity using Partial Directed Coherence (PDC) and Direct Transfer Function (DTF) from multivariate autoregressive (MVAR) models. The code simulates MVAR processes, fits model parameters via the Yule-Walker equations, and computes spectral connectivity measures, plotting the resulting frequency-domain relationships between multiple time series. It relies on NumPy, SciPy, Matplotlib, and FFT routines to implement the analysis pipeline, with functions for data generation, order selection via BIC, and spectral density estimation.

Mar 30, 2014