Skip to main content
SimilarExport ZIP
100%

Using tr

✓ Published0🌍 Public
LLeeMendelowitz
Last edited Feb 4, 2015
Created on Feb 4, 2015

This example demonstrates how to convert CSV files with non-Unix line endings into Unix format using the command-line utility `tr`. It shows two conversion scenarios: replacing carriage returns (`\r`) with line feeds (`\n`) for files with Mac OS 9 endings, and deleting carriage returns to handle Windows-style `\r\n` endings. The code processes the DC Metrorail ridership dataset, which contains daily totals, and verifies the results by comparing output files with `md5sum`. Alternative tools like `mac2unix` and `dos2unix` are also presented for the same task.

AI-generated description

Title: Fix the pesky "^M" carriage return control character using tr Date: 2015-02-04 3:20 Slug: remove-carriage-return-control-character Author: Lee Mendelowitz Tags: python Summary: How to fix the "^M" carriage return control character using tr

When working with Microsoft Excel 2011 on Mac OS X, it will export spreadsheets to comma separated .csv files or tab delimited .txt plain text files using the old Mac OS 9 line endings of a carriage return (\r). When viewed with less or vi, these characters appear as ^M, and the text file appears as one massively long line.

These end of line characters can be handled when importing data using R or Python's pandas library, but it's still a nuisance for other common Unix utilities like GNU's head, tail, less, wc -l, etc.

There are a few ways to convert these line endings using GNU utilities. You can use tr or dos2unix.

Using tr

Mac (\r) to Unix (\n)

To replace those ^M's with Unix line endings, use the tr utility:

tr '\r' '\n' < DC_Metrorail_Ridership.mac.csv > DC_Metrorail_Ridership.unix.csv

This will replace all carriage returns with line feeds.

Windows (\r\n) to Unix (\n)

Windows line endings use carriage return followed by line feed, so to convert Microsoft text files line endings to Unix use the following:

tr -d '\r' < DC_Metrorail_Ridership.windows.csv > DC_Metrorail_Ridership.unix.csv

Using dos2unix or mac2unix

Another option is to use the dos2unix utility. To install on Mac OS X using Homebrew:

brew update
brew install dos2unix

Mac (\r) to Unix (\n)

mac2unix -n DC_Metrorail_Ridership.mac.csv DC_Metrorail_Ridership.unix.csv

Windows (\r\n) to Unix (\n)

dos2unix -n DC_Metrorail_Ridership.windows.csv DC_Metrorail_Ridership.unix.csv

I made a GitHub gist where you can test all of this out. Clone the gist with:

git clone https://gist.github.com/7509ae7cfeaf1072822a.git

Similar vizzes

Loading thumbnail…

csv2tsv

This example converts comma-separated values to tab-separated values in a single pass. It parses a CSV file from disk using `d3.csv.parse` and immediately formats the resulting data structure as TSV with `d3.tsv.format`. The tool reads its input from a file path provided as a command-line argument and writes the converted output to standard output, enabling easy redirection. The code relies on the `d3` library and Node.js’s `fs` module for file access, and it is intended to be run as an executable script.

Mmbostock
56% match
Loading thumbnail…

truc

This example presents a simple SVG canvas and loads a CSV dataset using D3’s `csv` function, parsing each row’s `population` field into a number before logging the resulting array to the console. The visualization does not yet render any marks or interactions; it only demonstrates the initial data-loading and preparation step within a React-based ES6 project. All code runs against the D3 `select` and `csv` APIs, with the SVG dimensions read from the markup’s attributes.

NNanok09
54% match
Loading thumbnail…

Replace nested double quotes with single quotes in these CSV strings.

This example demonstrates a text-processing routine that cleans malformed CSV data by replacing nested double quotes with single quotes. It operates on two hardcoded strings containing street names and player nicknames with embedded quoted terms, such as `"Dearil "DJ""` and `"Calvert "C.J.""`. The code uses Ruby string interpolation with `%()` literals to define the input strings, then applies a regular expression to substitute the inner quote pairs. The output is not visualized but rendered as plain corrected CSV lines, showing how the transformation standardizes quote usage for downstream parsing.

CCaged
53% match
Loading thumbnail…

Public Transport Journeys by Type of Transport London Data

This example loads a CSV file of London public transport journey data from the London Data Store and prints a text-only summary of the dataset. The visualization directly displays the number of rows, columns, and kilobytes of the CSV, followed by the first 10 rows as a JSON string. Using the `d3.csv` function to parse the data, the program applies `d3.select` to append `<pre>` elements to the body, showing the raw metadata and sample records. The page features a green-on-black terminal aesthetic defined by custom CSS.

KKCachel
49% match
Loading thumbnail…

Gist fdfdd728007f46922fdc

This example demonstrates how to read and write text and CSV files in Ruby, using the CSV library. It reads names from a text file into an array, prints the array, and then appends the word "likes" to each name before saving the result to another text file. The code also writes the original names to a CSV file. This is a file-processing utility rather than a data visualization, focusing on data manipulation and storage.

AAndrelton
49% match
Loading thumbnail…

Loading and Parsing CSV Data

This example demonstrates how to load and parse CSV data into a usable format, displaying the result as a text summary of file size, row count, and column count. It uses the D3.js `csv` function to fetch and parse a remote CSV file of CSS named colors, then formats the parsed data with `d3.csvFormat`. The interface includes a file input for user-selected local CSV files, triggering the same parsing workflow via the `onchange` event handler.

Aatree-rohit
48% match
Loading thumbnail…

Loading and Parsing CSV Data

This example demonstrates loading and parsing a remote CSV file with D3 v7. It fetches the data using `d3.csv` and displays the formatted size, row count, and column count in a large `<pre>` element. The code also includes commented alternatives using `d3.csvParse` and the Fetch API for manual text processing. The data source is a GitHub Gist containing CSS named colors.

Ttrissentety
48% match
Loading thumbnail…

Trivikram - Fork of HTML Starter

This example demonstrates a simple HTML page that uses D3.js to load and display metadata about a CSV dataset, showing the dataset’s file size, row count, and column count in a styled message container. The page also embeds a static SVG image (a rabbit) and applies custom CSS for visual presentation. The code uses the `d3.csv` method to asynchronously fetch a CSV file from a GitHub gist URL, then updates the text content of the message container with the dataset’s details. The visualization highlights basic data loading and DOM manipulation with D3 v7, without any complex charts or interactive features.

Ttrivikramcheluri@gmail.com
48% match