Skip to main content
100%

A map-reduce class in Python, with the typical "hello, world!" word-counting example. You can download ulysses.txt file used in the example here: http://www.gutenberg.org/ebooks/4300

✓ Published0🌍 Public
Aadewes
Last edited Nov 21, 2013
Created on Nov 21, 2013

This example demonstrates a word-frequency analysis of James Joyce’s *Ulysses*, showing the 100 most common words and their counts. It uses a custom `MapReducer` class, built with Python’s `abc` module for abstract methods, to implement the map-reduce pattern. The `WordReducer` subclass defines a `map` method that tokenizes lines and tallies lowercase word occurrences, while `reduce` sums the counts. The `mapreduce` method groups mapped key-value pairs via `collections.defaultdict`. The final sorted output is printed with `pprint` from the text file downloaded from Project Gutenberg.

AI-generated description

Similar vizzes

Loading thumbnail…

counting things in javascript

A short JavaScript snippet counts word frequencies from a fixed string of text, using the array `reduce` method to tally occurrences and building a plain object that maps each word to its count. The example demonstrates a minimal functional programming pattern for aggregating categorical data, with the source data being a literal sentence split into an array of words. The output is logged to the console, showing a frequency distribution where repeated words like “pease” and “porridge” appear with higher counts than unique terms. No external libraries are used; the code relies solely on native JavaScript array and object methods.

Aamundo
58% match
Loading thumbnail…

Knuth/McIlroy word count task using Haskell

This example demonstrates a word-frequency analysis task using Haskell, showing how to compute and display the most common words from standard input. It chains data transformations through a pipeline: converting text to lowercase, replacing non-letter characters with newlines, splitting into words, and tallying counts with a hash map. The code relies on `Data.HashMap.Strict` for efficient key-value aggregation via `HashMap.fromListWith`, and uses `Data.List` for sorting and taking the top *n* results. The final output lists each word with its count, sorted in descending order of frequency. The example highlights functional composition and lazy evaluation in a concise, read-backwards data-flow style.

FFranklin Chen
57% match
Loading thumbnail…

Finding reduplicated words in a wordlist

This example scans a wordlist for reduplicated words—words whose letter pairs repeat—by analyzing bigram frequency. It computes the median frequency of all bigrams in each word and prints the word if that median equals 2, indicating a likely reduplication. The Python script uses `collections.Counter` to tally bigrams, a custom `median()` function, and `fileinput` to read lines from standard input. The output is a filtered list of words, showing a simple text-based visualization of the detected pattern.

Aamundo
53% match
Loading thumbnail…

Example from Programming4Humanists

This example visualizes word frequency in a text by processing the poem’s lines and filtering out common stopwords. It shows a ranked list of words by their occurrence count, displayed in descending order. The code uses XQuery 3.0 with functions like `fn:tokenize`, `fn:distinct-values`, and `fn:count` to compute frequencies, while reading the source text and stopword list from gist URLs. The rendering approach is a simple textual output of word elements, each carrying a `frequency` attribute, rather than a graphical chart.

CCliffordAnderson
52% match
Loading thumbnail…

Gist 6ab67c7133edab5b3b01760a3f226a25

This example compares the frequency of uppercase letters in the King James and Luther Bibles, printing a percentage of all uppercase letters per book. Using Python’s `re` module with custom regular expressions, it filters out lines with fewer than four words, skips verses’ opening formatting, and excludes all-uppercase words before counting letters. The output reports each translation’s uppercase share, revealing stylistic differences in capitalization. The script reads local text files (`kjv.txt`, `luther.txt`) and computes letter counts via `matchLetters` and `matchUCLetters` patterns.

HHoffmannP
49% match
Loading thumbnail…

Gist 4be75194a32c761f2a20

This example demonstrates how to use NLTK's frequency distribution object to analyze word frequencies in the Monty Python and the Holy Grail text (text6). The code imports the NLTK book module, creates a FreqDist from the text, and displays the top 50 word-frequency pairs, revealing unexpected results. The notebook then poses tasks for modifying the text and generating cumulative frequency plots, while also showing list comprehensions to filter words by length and frequency. The analysis relies on NLTK's FreqDist and the pre-loaded text corpus from the nltk.book module.

MMartiHearst
48% match
Loading thumbnail…

Gist 285672

This example demonstrates a language identification system by modeling text as bigram frequencies. It shows the most common two-letter sequences from a given input text, sorted by frequency. The Python code builds a bigram model using `ngrams` and `bigrams` functions, counts occurrences with a `defaultdict`, and removes rare bigrams below a threshold. It relies on the `collections` module and `operator.itemgetter` for sorting. The program reads a UTF-8 text file from the command line and prints the top ten bigrams with their counts.

Aamundo
48% match
Loading thumbnail…

JS Bin

This example counts character frequency in a long string of text and outputs a sorted array of characters with their occurrence counts. The code uses JavaScript's `reduce` method to build a frequency object, then sorts the result in descending order using a custom `compareNumbers` function. The data source is a hardcoded string of lowercase letters and underscores. The final output is logged to the console via `console.log`, showing the sorted array in descending order by frequency.

JJn1532
47% match