Skip to main content
100%

look up names of characters in text

✓ Published0🌍 Public
Aamundo
Last edited Jun 11, 2012
Created on Jun 11, 2012

This example maps the characters of UTF-8 text to their Unicode names. Reading input from standard input, the script decodes the text and iterates through each letter, using Python’s `unicodedata.name` to retrieve the official character name and printing it alongside the character, its code point in hexadecimal, and a Unicode escape sequence. Characters without a name are silently skipped. The output is written to standard output with UTF-8 encoding via `codecs.getwriter`.

AI-generated description

Similar vizzes

Loading thumbnail…

letters.py - command line tool to look up the name of Unicode characters in a text

The tool reads UTF-8 text from standard input and prints each character’s Unicode name, its code point in hexadecimal, and a JSON-style escape sequence. It uses Python’s `unicodedata.name` to look up names, `sys.stdin.read` to capture input, and `codecs.getwriter` to ensure UTF-8 output. Characters without a defined name are skipped silently. The output is displayed as a line per character, showing the letter, name, hex code, and escaped form.

Aamundo
78% match
Loading thumbnail…

Gist 3679500

This example demonstrates a Unicode character name lookup error and its resolution through Python’s `unicodedata` module. It shows that passing a two-character string like "ɬʼ" to `unicodedata.name` raises a `TypeError`, while each individual character—"ɬ" and "ʼ"—successfully returns its official Unicode name. The code uses the `unicodedata.name` API interactively in a Python shell, with the traceback illustrating the failure mode and the subsequent successful calls. The rendering approach is plain text output from the Python interpreter, highlighting the module’s requirement for a single-character input.

Aamundo
66% match
Loading thumbnail…

Gist 288417

This visualization displays the frequency of every character in a text file, sorted from least to most common. The Python script reads the file’s contents, counts each Unicode character’s occurrences in a dictionary, then prints the character alongside its count. It uses the `codecs` module to handle UTF-8 output and standard file reading to load the source text. The result is a plain textual ranking, not a graphical chart, with each line showing a character and its frequency.

Aamundo
65% match
Loading thumbnail…

futzing with python encodings.

This example demonstrates how Python 2.7 handles Unicode strings when serializing to and from JSON. It shows a dictionary of Ethiopic syllable names being written to a file with `json.dumps`, which escapes the characters into `\u1234`-style sequences, and then reloaded with `json.load` to recover the original Unicode objects. The process highlights the round-trip fidelity of the `json` module, even though the intermediate file appears “escape-y,” and the code also uses `unicodedata.name` to fetch character names.

Aamundo
54% match
Loading thumbnail…

Gist 285672

This example demonstrates a language identification system by modeling text as bigram frequencies. It shows the most common two-letter sequences from a given input text, sorted by frequency. The Python code builds a bigram model using `ngrams` and `bigrams` functions, counts occurrences with a `defaultdict`, and removes rare bigrams below a threshold. It relies on the `collections` module and `operator.itemgetter` for sorting. The program reads a UTF-8 text file from the command line and prints the top ten bigrams with their counts.

Aamundo
54% match
Loading thumbnail…

[注音符号字母转换] #string

This example demonstrates how to normalize Unicode text that contains accented or composite characters into plain ASCII, showing the transformation before and after processing. It uses Python’s `unicodedata` module, specifically the `normalize` function with `NFKD` and `NFD` forms, followed by encoding to ASCII with the `'ignore'` error handler, or by using a dictionary of combining characters with `translate` to strip diacritical marks. The code includes two methods: one that removes non-ASCII characters entirely, and another that preserves them while eliminating accent marks, with comments explaining the behavior for both Python 2 and 3.

SShawn91
53% match
Loading thumbnail…

[print_unicode] python2 下在命令行中正常打印嵌套在 dict/list 内的 unicode 字符 #python #string

This example shows a Python 2 utility function that makes nested Unicode characters inside dictionaries and lists print correctly to the command line. The function `print_uni` takes any string `s`, applies `repr` to capture its escaped form, and then decodes that representation using `"unicode-escape"` to reveal the actual Unicode glyphs. It relies on the built-in `repr` function and the string `decode` method, demonstrating a workaround for Python 2’s default ASCII-based printing of container structures. The code is a standalone snippet from a gist by Shawn91, focusing on a common debugging pain point.

SShawn91
51% match
Loading thumbnail…

charsethero : convert mixed utf-8 and iso-latin content to utf-8

This example demonstrates a command-line Perl script that converts mixed UTF-8 and ISO 8859-1 encoded text into a single UTF-8 stream. It reads from standard input, processes each line, and outputs the converted result. The core logic uses regular expressions to identify multi-byte UTF-8 sequences and single-byte Latin-1 characters, then applies the `encode_char_utf8` function to re-encode any non-UTF-8 bytes. The script also includes a `force_latin` function for reverse conversion, though the main pipeline calls `force_utf8`. The data source is standard input, making it a filter for text streams.

PPhilippe Rivière
51% match