Skip to main content
SimilarExport ZIP
100%

Trivia: Generate primes between 1 and 100 million with Hadoop MapReduce job. Inspired by https://gist.github.com/krishnanraman/6346307.

✓ Published0🌍 Public
PP7h
Last edited Apr 13, 2014
Created on Apr 12, 2014

This example compares Hadoop MapReduce and pure Java for computing prime numbers between 1 and 100 million. It uses the Hadoop `Job`, `Mapper`, `TextInputFormat`, and `TextOutputFormat` APIs to process a file with one number per line, emitting each prime via `context.write`. The MapReduce job runs in about 2 minutes 20 seconds, while a pure Java loop takes 5 minutes 26 seconds. The output file, `part-r-00000`, contains 5,761,455 primes, demonstrating the parallel framework’s speed advantage over a single-threaded approach.

AI-generated description

Similar vizzes

Loading thumbnail…

four variations of isprime for the first thousand integers, shuffled and rotated counts don't include original in their counts, but require it

Four variations of the `isprime` function for the first thousand integers are compared, showing how primality changes under digit shuffling, rotation, and reversal. The visualization plots each integer along the x-axis, with rows for the original prime status, the count of shuffled permutations that are prime, the count of rotated permutations that are prime, and whether the reversed digits are prime. The data comes from a CSV file processed with d3.js, using SVG elements to render the scatter-style chart. It highlights which primes remain prime under these digit transformations and which do not.

BBBischof
52% match
Loading thumbnail…

Gist 63d8bf6ae5ef850809c429573e052edc

This example demonstrates the computation of the 8181st Fibonacci number using an iterative method with Java’s `BigDecimal` class to handle arbitrarily large integer results. The code defines two functions: a naive recursive version that would be impractical for large inputs, and an optimized iterative loop that builds the sequence by adding the previous two values. The main method prints the final result for 8181, showcasing how `BigDecimal` enables precise arithmetic beyond standard primitive type limits.

HHsuHsiaoHsuan
48% match
Loading thumbnail…

Some data structures and algorithms in Java

This example implements classic data structures and algorithms in Java, including binary search trees, merge sort, quicksort, factorial, exponentiation, Fibonacci sequences, string permutation generation, and range unionization. The code uses Java’s built-in `ArrayList`, `Arrays`, and `Collections` APIs, along with custom classes like `Node`, `Tuple`, and `Unionizer`. Random test arrays are generated via `Math.floor` and `Math.random` to benchmark sorting performance, with results printed to the console. The visualization highlights the logic behind each algorithm through clear method signatures and step-by-step control flow.

KKirkHunter
47% match
Loading thumbnail…

Writing 10M lines of a query result to a table in Power Query via R

This example demonstrates how to write 10 million rows of a query result to a CSV file from Power Query by invoking an R script. It shows the integration of Power Query’s table generation with R’s file-export capabilities, specifically using the `R.Execute` function to run `write.csv` on a dataset. The Power Query code creates a list of numbers from 1 to 10,000,000 via `{1..10000000}`, converts it to a table with `Table.FromList`, and passes that table as the `dataset` parameter to the R script. The R script then writes the data to `d:\Downloads\out.csv`.

HHugoberry
45% match
Loading thumbnail…

Gist c0ec6bb768fd2672307b0807e4ee46fb

This example benchmarks four JavaScript loop styles—a standard `for` loop, a reversed `for` loop, a `for…of` loop, and `forEach`—on a fixed array of ten numbers. It measures execution time by repeating each loop a thousand times via `performance.now()` timestamps, accumulating the total milliseconds. The code defines each loop as a separate function and calls them through a `test` helper, logging the sum to the console. The comparison uses a simple in-memory data array, with no external libraries, focusing on raw loop performance in a JavaScript environment.

MMAKIO135
45% match
Loading thumbnail…

Gist 740340

This example demonstrates a parallelized aggregate operation that sums the integers from zero to one million using Language-Integrated Query (LINQ). The code uses the `Enumerable.Range` method to generate the data sequence and calls `AsParallel()` to enable parallel execution across multiple threads. The `Aggregate` function then combines each element with an accumulator, applying the addition lambda `(x, y) => x + y` to produce the total. The visualization likely maps the resulting sum or the execution process, though the rendering method is not specified in the provided source code.

RRestuta
45% match
Loading thumbnail…

Helpers for TDunnings Java TDigest library

This example demonstrates how to use the TreeDigest implementation from the Java TDigest library to compute approximate cumulative distribution functions (CDFs) and probability density functions (PDFs) from large datasets. It shows three Scala workflows: serializing and deserializing digests, merging per-element digests across an Apache Spark RDD using `reduce`, and using Kryo serialization to pass TreeDigest objects directly between Spark tasks. The code samples from exponential and normal distributions via Apache Commons Math, then compares CDF values computed locally versus through Spark, highlighting the library’s integration with distributed data processing.

RRobColeman
44% match
Loading thumbnail…

Gist 737033

This example demonstrates a MapReduce aggregation over a MongoDB collection of crime records, showing the count of crimes grouped by property name. The code first seeds a `crimes` collection with sample data, then defines a `map` function that emits each property name with a value of 1, and a `reduce` function that sums the emitted values. The MapReduce operation runs via `db.crimes.mapReduce()` and outputs the results as a JSON array using `printjson`. It uses MongoDB’s JavaScript shell API for data insertion, mapping, reduction, and result retrieval.

CCaged
44% match