charsethero : convert mixed utf-8 and iso-latin content to utf-8

100%

charsethero : convert mixed utf-8 and iso-latin content to utf-8

✓ Published0🌍 Public

Last edited Jun 8, 2014

Created on Jun 8, 2014

This example demonstrates a command-line Perl script that converts mixed UTF-8 and ISO 8859-1 encoded text into a single UTF-8 stream. It reads from standard input, processes each line, and outputs the converted result. The core logic uses regular expressions to identify multi-byte UTF-8 sequences and single-byte Latin-1 characters, then applies the `encode_char_utf8` function to re-encode any non-UTF-8 bytes. The script also includes a `force_latin` function for reverse conversion, though the main pipeline calls `force_utf8`. The data source is standard input, making it a filter for text streams.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist f5904c4a8279840f341742d30c5d9a9e

This example demonstrates a PowerShell script that processes a text file by replacing caret characters with semicolons and writes the result to a new file. It shows a batch text-transformation workflow using the `Get-Content` cmdlet to read input in chunks of 1000 lines, a custom filter function `replace-chars` that applies the `-replace` operator, and `Add-Content` to append the processed lines to the output file. The script first checks if the output file exists and clears it to ensure a fresh write. The approach highlights pipeline-based file handling and character encoding preservation without additional libraries.

HHugoberry

55% match

Loading thumbnail…

Normalizer

This example demonstrates a string normalization function that converts accented and special characters into plain ASCII equivalents. It builds a character mapping from two parallel strings, then processes each input character, replacing matches with their mapped counterparts and lowercasing the final result. The implementation relies on standard JavaScript string methods such as `charAt`, `hasOwnProperty`, `join`, and `toLowerCase`. The code is presented as a standalone JavaScript function, with no external libraries or data sources involved, showing the raw logic behind text sanitization in a compact form.

LLuisSevillano

51% match

Loading thumbnail…

Using tr

This example demonstrates how to convert CSV files with non-Unix line endings into Unix format using the command-line utility `tr`. It shows two conversion scenarios: replacing carriage returns (`\r`) with line feeds (`\n`) for files with Mac OS 9 endings, and deleting carriage returns to handle Windows-style `\r\n` endings. The code processes the DC Metrorail ridership dataset, which contains daily totals, and verifies the results by comparing output files with `md5sum`. Alternative tools like `mac2unix` and `dos2unix` are also presented for the same task.

LLeeMendelowitz

46% match

Loading thumbnail…

Pig Latin Converter

This visualization demonstrates how an English sentence is transformed into Pig Latin, showing the word-by-word conversion process. It displays the original sentence "I speak Pig Latin" alongside its Pig Latin equivalent by applying phonetic rules to each word. The code uses the XQuery 3.0 string-join function and fn:tokenize to split the input, then iterates through each word with a for loop. It checks the first letter against a list of vowels using fn:lower-case and fn:substring, then concatenates the word parts with the "||" operator to form the translated output. The rendering approach is purely text-based, with no graphical or interactive elements.

CCliffordAnderson

45% match

Loading thumbnail…

Read File or HTTP

This example demonstrates a small utility for reading text content from either a local file or a remote HTTP URL, abstracting the source so callers can use a single `get` function. The code shows a Node.js script that checks the URL protocol: if it starts with `http:`, it uses the `http.get` method to stream the response, collecting data chunks and joining them as a UTF-8 string; otherwise, it falls back to `fs.readFile` with a `"utf8"` encoding. Both paths invoke a callback with either an error or the resulting body, making the helper suitable for loading data files or web resources in server-side visualizations.

Mmbostock

44% match

Loading thumbnail…

Gist 75454

This example demonstrates string manipulation utilities in Objective-C, converting camelCase or mixed-case strings into either underscore-separated or dash-separated formats. The code defines an NSString category with `underscore` and `dasherize` methods, which internally call a private `normalizeWith:` method that iterates over each character, detects uppercase letters and existing delimiters, and inserts the chosen separator while preserving the original text. The implementation uses NSCharacterSet, NSMutableString, and unichar APIs to perform character-by-character processing, ultimately returning a lowercase version of the transformed string.

CCaged

43% match

Loading thumbnail…

Pandoc bash conversion

This visualization demonstrates a user’s workflow of converting and renaming files across directories using shell commands. It shows three distinct bash operations: deleting HTML files in a specific folder, converting sociology files to DOCX via Pandoc, and renaming XML files to HTML using the `rename` utility. The code relies on standard shell scripting constructs and the Pandoc command-line tool, with no graphical or interactive rendering libraries like D3 or Three.js. The data source is the local file system paths referenced in each command, and the rendering approach is purely textual—showing the raw bash scripts rather than any visual output.

CCliffordAnderson

43% match

Loading thumbnail…

Gist 44be7198b7913c5e683122ab224baa9e

This example demonstrates a shell-based spinner that provides visual feedback during a long-running command-line task. Instead of a graphical chart, it renders a rotating line animation directly in the terminal by updating a single carriage-returned character (`\r`) on the standard error stream. The script iterates through the characters of the string `/|-\` using a modulo index, printing one frame per line of piped input. It relies on standard shell constructs like `printf`, a `while` loop with `read`, and an environment variable for state, with no external JavaScript libraries or data sources involved. The spinner stops and clears the line after the input stream ends.

HHerst

43% match