Skip to main content
SimilarExport ZIP
100%

A fast, serializable, Scala implementation of tDigest. Thrown together for a Spark project.

✓ Published0🌍 Public
RRobColeman
Last edited Jan 28, 2016
Created on Jan 27, 2016

This visualization demonstrates a serializable Scala implementation of the tDigest algorithm, designed for fast approximate quantile estimation in Apache Spark. It defines `Centroid` and `TDigest` case classes that use Scala’s mutable `TreeSet` for sorted centroid storage, with methods for updating, merging, compressing, and computing cumulative distribution and inverse CDF values. The code relies on `org.apache.spark.SparkContext` and `org.apache.commons.math3.distribution.ExponentialDistribution`, using `scala.util.Random` for centroid shuffling during merges.

AI-generated description

Similar vizzes

Loading thumbnail…

Helpers for TDunnings Java TDigest library

This example demonstrates how to use the TreeDigest implementation from the Java TDigest library to compute approximate cumulative distribution functions (CDFs) and probability density functions (PDFs) from large datasets. It shows three Scala workflows: serializing and deserializing digests, merging per-element digests across an Apache Spark RDD using `reduce`, and using Kryo serialization to pass TreeDigest objects directly between Spark tasks. The code samples from exponential and normal distributions via Apache Commons Math, then compares CDF values computed locally versus through Spark, highlighting the library’s integration with distributed data processing.

RRobColeman
76% match
Loading thumbnail…

An Approximate Distribution wrapper class for TDunnings java TDigest library

This example demonstrates how to estimate and visualize approximate probability distributions from large datasets using a wrapper class around TDunning's Java TDigest library. The code shows both local and Spark-based aggregation paths, where data is converted into TreeDigest instances and merged via reduction operations. It generates CDF and PDF sequences by sampling support points between observed minimum and maximum values, then plots these distributions for comparison. The example also illustrates Kryo serialization configuration for Spark clusters, using 100,000 samples drawn from a normal distribution to validate the approach.

RRobColeman
73% match
Loading thumbnail…

Gist 9ea11dbcc52b4bb1c048

This example shows performance analysis for a data engineering capstone project, comparing a legacy Python log parser against a proposed Scala implementation on Spark and Mesos. The visualization tracks transfer metrics from ASCP log data, specifically examining throughput and packet rates. Using d3.js for charting, it evaluates how efficiently the new streaming solution handles regex parsing of unstructured logs. The project also demonstrates integration with Redis for persistent storage and FaspStream for live data ingestion, with the D3 visualizations presenting benchmark results from test log datasets.

BBBischof
50% match
Loading thumbnail…

Gist 4945314

This example demonstrates a simple yet effective visualization of data transformation in Scala, highlighting implicit conversions between mutable and immutable collection types. The code creates a mutable `ListBuffer[Double]`, appends two numeric values, and then passes it to a function expecting an immutable `List`. Through an implicit conversion, the `ListBuffer` is seamlessly converted to a `List`, allowing the operation to succeed. The visualization maps the underlying mechanics of this type conversion, showing the relationship between the original buffer and its converted form. It leverages Scala’s standard collection library and implicit resolution to illustrate the flow of data between collection types.

RRCura
49% match
Loading thumbnail…

The d3 test

This scatter chart plots contig length against GC content from a FASTA file. The code parses sequence headers using `d3.dsvFormat` to extract numeric values, then renders circles with `d3.scaleLinear` and `d3.axisBottom`/`d3.axisLeft`. The visualization uses hardcoded sample data as a fallback, with axes and points styled via CSS.

LLaurynKW
49% match
Loading thumbnail…

Gist 1dcc8a9bd27317b89fdde2c22c049100

This example presents a Scala-based feature engineering pipeline for machine learning models, specifically for predicting eCPM (effective cost per mille) in advertising. The code defines a system of feature computers that transform prediction requests into sparse feature vectors using Apache Spark's MLlib library (`SparkVectors.sparse`) and the `MurmurHash3` hashing function. It shows how categorical features like app publisher and model identifiers are hashed into indexed blocks within a numeric vector, with each block's position and size tracked through `FeatureComputerParams`. The implementation uses pattern matching in `FeatureComputerRouter` to select appropriate computations and aggregates values into a map that can be converted to Spark's sparse vector format for model training.

RRobColeman
48% match
Loading thumbnail…

Distribution

Four small multiples display transaction and login distributions from a scored features CSV. D3 v7 renders SVG histograms showing log-transformed transaction amounts, credit versus debit volume, channel breakdowns, and login attempt counts. The interface includes exact-match and multi-select filters plus a checkbox to flag anomalies above the 95th percentile. Interactive controls update all charts simultaneously.

SSreedevi Itta
48% match
Loading thumbnail…

This is the result of JODA not being serializable in scala, resorting to utter bullshit, assumes year is 2016.

This visualization charts time-series data using a custom timestamp conversion workaround. The chart maps dates and times to Unix epoch integers through a Scala function that hardcodes the year as 2016, manually calculating day-of-year offsets and time-of-day seconds. The function parses month abbreviations via string indexing, converts day strings to integers, and splits time strings into hours, minutes, and seconds. This approach bypasses JODA datetime serialization limitations, producing numeric x-axis values for plotting. The rendered graphic likely uses a standard plotting library, though the code excerpt only shows the formatter.

BBBischof
48% match