explicit encoding declaration avoids conflicts for non-ASCII chars

100%

explicit encoding declaration avoids conflicts for non-ASCII chars

✓ Published0🌍 Public

Last edited May 27, 2014

Created on Jul 23, 2013

This Ruby script demonstrates a text-cleaning workflow for BibTeX files, showing how to systematically remove unwanted metadata fields, replace non-ASCII characters with LaTeX-safe escapes, and strip problematic symbols. It reads a bibliography file, processes each line while tracking multi-line field deletions, and applies regular-expression-based replacements for characters like ó, é, and ö. The script uses Ruby’s File I/O, Regexp, and string mutation methods (`gsub!`), with a debug flag to inspect specific entries. The output is a fixed bibliography file ready for LaTeX compilation, preserving entry structure while sanitizing content.

AI-generated description

Similar vizzes

Loading thumbnail…

charsethero : convert mixed utf-8 and iso-latin content to utf-8

This example demonstrates a command-line Perl script that converts mixed UTF-8 and ISO 8859-1 encoded text into a single UTF-8 stream. It reads from standard input, processes each line, and outputs the converted result. The core logic uses regular expressions to identify multi-byte UTF-8 sequences and single-byte Latin-1 characters, then applies the `encode_char_utf8` function to re-encode any non-UTF-8 bytes. The script also includes a `force_latin` function for reverse conversion, though the main pipeline calls `force_utf8`. The data source is standard input, making it a filter for text streams.

PPhilippe Rivière

53% match

Loading thumbnail…

Gist c994b5a229194ab2c98a8580d48f54ef

This example shows a Perl script that corrects FASTA sequence headers by replacing pipe characters with hyphens, making the files compatible with the CEGMA genome analysis tool. The script reads an input FASTA file and writes the corrected output to a second file. It uses the standard Perl file I/O operations `open`, `close`, and the regex substitution operator to modify only the header lines that begin with `>[A-Z]{3}`. The script also checks for an existing output file to prevent accidental overwriting, and it prints a usage message if the correct number of arguments is not provided.

NNpfn

51% match

Loading thumbnail…

Replace nested double quotes with single quotes in these CSV strings.

This example demonstrates a text-processing routine that cleans malformed CSV data by replacing nested double quotes with single quotes. It operates on two hardcoded strings containing street names and player nicknames with embedded quoted terms, such as `"Dearil "DJ""` and `"Calvert "C.J.""`. The code uses Ruby string interpolation with `%()` literals to define the input strings, then applies a regular expression to substitute the inner quote pairs. The output is not visualized but rendered as plain corrected CSV lines, showing how the transformation standardizes quote usage for downstream parsing.

CCaged

51% match

Loading thumbnail…

use make to build Latex/Xelatex documents. Biblatex support included.

This makefile example demonstrates how to automate the compilation of LaTeX and XeLaTeX documents, including full bibliography support through BibLaTeX and biber. It shows separate build targets for standard LaTeX and XeLaTeX workflows, each with and without bibliography processing, as well as a clean target to remove auxiliary files. The makefile uses the `pdflatex` and `xelatex` commands with specific flags for synctex and nonstopmode, while managing build artifacts in a hidden `.build` directory. It copies source and `.bib` files into the build directory before running the appropriate compiler sequence, then moves the final PDF back to the project root. The code relies on standard Unix utilities like `mkdir`, `cp`, `mv`, and `rm` within Make’s recipe syntax.

CChristosT

51% match

Loading thumbnail…

Gist f5904c4a8279840f341742d30c5d9a9e

This example demonstrates a PowerShell script that processes a text file by replacing caret characters with semicolons and writes the result to a new file. It shows a batch text-transformation workflow using the `Get-Content` cmdlet to read input in chunks of 1000 lines, a custom filter function `replace-chars` that applies the `-replace` operator, and `Add-Content` to append the processed lines to the output file. The script first checks if the output file exists and clears it to ensure a fresh write. The approach highlights pipeline-based file handling and character encoding preservation without additional libraries.

HHugoberry

51% match

Loading thumbnail…

XQuery Blue Mountain Project - Part III

This example shows a pipeline for transforming a single TEI XML document from the Blue Mountain Project—a digitized 1920 issue of the French journal *Action*—into clean, article-level TEI files. The first script removes facsimile elements, end-hyphenation markers, and line breaks, replacing `<lb>` elements with spaces. A second script extracts metadata and article content into a fresh TEI template, while a third builds individual TEI files for each article. The code uses XQuery 3.1 with the `fn:doc` API for document access and node operations like `copy`, `modify`, and `replace node`.

CCliffordAnderson

51% match

Loading thumbnail…

How to deal with renaming invalid UTF8 characters in directories or files

This example demonstrates a command-line workflow for detecting and renaming files with invalid UTF-8 characters, likely originating from latin-1 (ISO-8859-1) encodings. It first uses a Perl one-liner with a regular expression to filter filenames containing non-UTF-8 byte sequences, then applies the `rename` utility with Perl’s `Encode` module to convert those names from latin-1 to UTF-8. The process also references `iconv` and `recode` for verification. The solution relies on Unix shell scripting, piping `find` output through the filters, and handling character encoding transformations at the filesystem level.

TThomasG77

50% match

Loading thumbnail…

Gist 4127915bd94a1fc0518bb0cceea3b612

This example shows a Perl script that filters complete CEGs (Core Eukaryotic Genes) from a CEGMA output FASTA file, extracting only sequences corresponding to KOG identifiers not listed as missing in the completeness report. It reads three input files—the CEGMA FASTA, a completeness cutoff table, and the completeness report—then writes a new FASTA file. The script uses Perl’s built-in file handling and regular expression matching to parse KOG IDs, track a flag for sequence inclusion, and output the selected sequences.

NNpfn

50% match