Gist 285672

100%

✓ Published0🌍 Public

Last edited Jan 25, 2010

Created on Jan 25, 2010

This example demonstrates a language identification system by modeling text as bigram frequencies. It shows the most common two-letter sequences from a given input text, sorted by frequency. The Python code builds a bigram model using `ngrams` and `bigrams` functions, counts occurrences with a `defaultdict`, and removes rare bigrams below a threshold. It relies on the `collections` module and `operator.itemgetter` for sorting. The program reads a UTF-8 text file from the command line and prints the top ten bigrams with their counts.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist 288417

This visualization displays the frequency of every character in a text file, sorted from least to most common. The Python script reads the file’s contents, counts each Unicode character’s occurrences in a dictionary, then prints the character alongside its count. It uses the `codecs` module to handle UTF-8 output and standard file reading to load the source text. The result is a plain textual ranking, not a graphical chart, with each line showing a character and its frequency.

Aamundo

71% match

Loading thumbnail…

Gist 1305262

This visualization maps the distribution of Besemah-language text lines extracted from a source file. It uses Python’s `collections.Counter` to count the occurrence of each line, then renders the data as a visual plot, likely showing frequency or positional patterns. The code reads a UTF-8 encoded `Texts.txt`, splits it into lines, and filters for lines starting with `\tx` (excluding the bare marker), building a list of Besemah lines for analysis. The rendering approach is not specified in the visible code, but the example demonstrates text-data parsing and counting using standard Python libraries.

Aamundo

63% match

Loading thumbnail…

Gist 7518328

This example computes and displays the most relevant terms (including single words, bigrams, and trigrams) from a pair of documents by calculating term frequency–inverse document frequency (TF-IDF). It first tokenizes Portuguese text using NLTK’s `RegexpTokenizer` and removes stopwords, then builds a vocabulary of unigrams, bigrams, and trigrams using `nltk.bigrams` and `nltk.trigrams`. For each document, the code outputs each term’s raw frequency, TF, IDF, and TF-IDF score, and finally prints a ranked list of the highest scoring terms across all documents. The script relies solely on Python’s `math` module and NLTK, with no external visualization library.

TTheLady

61% match

Loading thumbnail…

Gist 2116a7bda7ef9a6364f3

This example demonstrates how to use NLTK’s frequency distribution object to analyze word frequencies in the pre‑loaded text of Monty Python and the Holy Grail. It first displays the top 50 most frequent words, then guides the viewer through tasks that involve examining the raw text, modifying it, and rerunning the frequency calculation. A cumulative frequency plot is generated, and a list comprehension isolates words longer than five characters that occur at least five times, sorted alphabetically. The code relies on NLTK’s `FreqDist` and the `nltk.book` collection.

MMartiHearst

60% match

Loading thumbnail…

Gist 4be75194a32c761f2a20

This example demonstrates how to use NLTK's frequency distribution object to analyze word frequencies in the Monty Python and the Holy Grail text (text6). The code imports the NLTK book module, creates a FreqDist from the text, and displays the top 50 word-frequency pairs, revealing unexpected results. The notebook then poses tasks for modifying the text and generating cumulative frequency plots, while also showing list comprehensions to filter words by length and frequency. The analysis relies on NLTK's FreqDist and the pre-loaded text corpus from the nltk.book module.

MMartiHearst

60% match

Loading thumbnail…

Gist 8583392

This example displays the frequency of personal pronouns in the Brown Corpus news category as a crude text bar chart. Using NLTK, it loads the news texts, tokenizes them, and applies `FreqDist` to count lowercase pronoun occurrences. The output prints each pronoun followed by a row of asterisks, where every asterisk represents 20 occurrences, alongside the raw count. The visualization intentionally uses a simple, non-graphical format, showing relative frequencies such as “he” dominating with 642 mentions and “us” appearing only 13 times.

Aamundo

57% match

Loading thumbnail…

Finding reduplicated words in a wordlist

This example scans a wordlist for reduplicated words—words whose letter pairs repeat—by analyzing bigram frequency. It computes the median frequency of all bigrams in each word and prints the word if that median equals 2, indicating a likely reduplication. The Python script uses `collections.Counter` to tally bigrams, a custom `median()` function, and `fileinput` to read lines from standard input. The output is a filtered list of words, showing a simple text-based visualization of the detected pattern.

Aamundo

57% match

Loading thumbnail…

Gist ec04ecc64fcf9189226ee0e77d2f14a1

This example demonstrates a text-to-NATO-alphabet converter. It takes user input—either from command-line arguments or an interactive prompt—and maps each alphanumeric character to its corresponding NATO phonetic word, separating them with hyphens. The output displays the fully spelled phonetic sequence, including a special `[SPACE]` token for spaces and bracketed fallbacks for unsupported symbols. The implementation uses a Python dictionary for the phonetic mapping and the `sys.argv` list for command-line input handling.

Aant1j

56% match