Skip to main content
100%

Gist 6c1ae0e4fff5fb8d4704

✓ Published0🌍 Public
Aandyreagan
Last edited Aug 21, 2015
Created on Aug 21, 2015

This example shows a text-parsing utility that cleans and tokenizes a raw string into lowercase words. It first replaces common punctuation sequences like dashes and double quotes with spaces, then uses a regular expression with Unicode flags to extract word tokens, allowing characters such as `@`, `#`, `&`, and hyphens. The code relies on Python’s standard `re` module and basic string methods, with no external visualization libraries or rendering pipeline. The output is a list of normalized words, illustrating a preprocessing step rather than a visual chart or graph.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist 55fc864ec9cea1107845d3ed7a3dab8b

This example demonstrates a text-cleaning pipeline for caption data, showing how raw captions are converted into tokenized, filtered lists for downstream analysis. The visualization itself is not rendered; instead, the focus is on the data preprocessing logic. The code uses `pandas` with `apply` to run natural language processing via `spaCy` (`nlp`), extracting tokens from the `Caption` column. It then applies a custom `tokens` function that removes stop words and non-alphanumeric characters using `re.match`, while preserving tokens containing "@" symbols. The result is a new `clean_caption` column, highlighting a common step in text mining workflows.

AArizonaTay
65% match
Loading thumbnail…

Gist 641fc44c27c66ee9ef09

This example demonstrates a Latin text preprocessing pipeline that converts raw words into lemmata and macronized dictionary entries. It shows how a sample sentence like "quo usque tandem abutere catilina nostra patientia" is tokenized, cleaned of punctuation, and processed word-by-word. The code uses the CLTK `LemmaReplacer` for lemmatization, lxml’s `etree` to parse the Perseus Latin lexicon XML file, and custom string manipulation to transform entries like "a^bu_tor" into "abūtor". The workflow defines classes for text and word-level morphological analysis, though the dictionary lookup for full texts runs too slowly to complete.

TTylerKirby
63% match
Loading thumbnail…

Gist 1f915686384da18e413f6fc5f50abfc7

This example visualizes a data-transformation utility that removes dashes from string arrays. It demonstrates a reusable JavaScript function called `removeDashesFrom`, which iterates through an input array and strips every hyphen character from each element using the `replace` method. The code logs its execution to the console, indicating a step-by-step processing flow. While no specific visualization library is used, the example highlights a common data-cleaning operation, likely intended for preprocessing datasets before feeding them into a charting framework like D3.js or Three.js. The function returns the modified array, making it suitable for integration into larger data-visualization pipelines where clean string formatting is required.

WWill Carter
62% match
Loading thumbnail…

A simple depunctuator with and without regexes

This example demonstrates a simple text-cleaning function that removes unwanted characters from strings, using two sample sentences to show the before-and-after effect. The code defines a `depunctuate` function in Python, which iterates through each character of the input and builds a new string by excluding any letters found in a provided `badletters` set. It uses basic string concatenation and a `for` loop, with no external libraries. Two print statements call the function: the first strips punctuation like periods and exclamation marks, while the second removes digits and commas from a numeric phrase.

Aamundo
61% match
Loading thumbnail…

Gist 6437893f183d99e863d835425ffa46f8

This example visualizes the distribution of tokenized caption text from a dataset of images, showing how frequently each word appears after cleaning. The visualization displays the processed terms as a bar chart or word cloud, with the data derived from the `Caption` field of the input dataframe. The preprocessing pipeline loads the `en_core_web_lg` spaCy model to tokenize each caption, removes English stop words and non-alphanumeric characters, and converts all terms to lowercase. The code relies on the `spacy` library for linguistic analysis and standard Python regex operations via the `re` module to filter tokens, while the `stopwords` set comes from NLTK.

AArizonaTay
61% match
Loading thumbnail…

Gist 2e867d04c508a033f4e88e66d1c3649a

This example shows a string-processing algorithm that removes repeated characters from an input word. The code defines a function, `removeRepeatedChars`, which iterates through the original string and builds a new string by appending characters only when they do not already appear in the newly constructed result. It uses nested loops to compare each character against the current new word and tracks matches with a counter variable. The implementation relies on standard C++ string operations and does not use any external visualization libraries, instead presenting the logic directly as source code.

MMistyBlunch
60% match
Loading thumbnail…

Gist 614119

This example processes interlinear linguistic annotation from standard input, splitting each line into words and glosses to build a structured data array. It uses Node.js `process.openStdin()` and `stdin.on('data')` to read chunks, with a `trim` function to clean whitespace and `extractLines` to split text by newlines. The code logs each line to the console, demonstrating a simple text-parsing workflow for linguistic data.

Aamundo
60% match
Loading thumbnail…

Python Utilities for Tweets

This utility module processes raw tweet JSON objects for text analysis. It extracts timestamps via `datetime.strptime`, hashtags, user mentions, and cleaned text by stripping URLs, mentions, and media using entity indices. It further sanitizes punctuation with the `string` module, removes English stopwords from `nltk.corpus`, and applies Lancaster stemming via `nltk.stem.lancaster.LancasterStemmer`. The code demonstrates reusable functions for preparing Twitter data, relying solely on standard Python and NLTK libraries without any visualization or rendering components.

Aabehmiel
60% match