Gist 6791729

100%

✓ Published0🌍 Public

Last edited Sep 13, 2013

Created on Oct 2, 2013

This example demonstrates an efficient pipeline for importing a large CSV dataset into a LevelDB key-value store. It streams the 1994.csv file (5.2 million lines) through `binary-split` to separate lines without creating JavaScript objects, then groups them into 16 MB batches via `byte-stream`. Each batch is written using LevelDB's batch API (`db.batch()`), with lines stored as raw binary values under incrementing integer keys. The process avoids object allocation entirely and completes the import in roughly 50 seconds.

AI-generated description

Similar vizzes

Loading thumbnail…

level-prefix bug

The example demonstrates a round-trip file storage and retrieval process through a networked LevelDB database, verifying that a streamed file survives the write-read cycle unchanged. It uses `level-prefix` to namespace all keys under an `attachments-` prefix, while `multilevel` exposes the database over HTTP and `level-store` provides streaming read/write operations. The code reads the current source file, pipes it into `store.createWriteStream`, then reads it back with `store.createReadStream` and compares the original bytes against the concatenated chunks using Node’s `assert`.

Aalessioalex

62% match

Loading thumbnail…

Gist 585199

This visualization loads an external CSV file and parses its rows into JavaScript objects, mapping each column name from an array to the corresponding values in each line. It handles quoted fields that contain commas by merging split segments until the quote count is even, and it logs warnings for malformed lines that do not match the expected number of columns. The example demonstrates a reusable CSV-asynchronous-loading helper, but does not show any charting, layout, or rendering code beyond the data preparation stage.

RRandomEtc

60% match

Loading thumbnail…

CVS to JSONL example

This example shows a practical data-transformation workflow that converts a CSV file into JSONL format, producing one JSON object per line. It fetches a remote CSV file from a GitHub gist using the `fetch:text` function, parses it into records with `csv:parse`, and iterates through each record to build key-value pairs. The code distinguishes between integer and string values using `castable as xs:integer`, then constructs JSON maps with `xml-to-json` using XQuery 3.1. The output is unindented JSONL, suitable for streaming or line-oriented processing.

CCliffordAnderson

57% match

Loading thumbnail…

fresh block

This example demonstrates a minimal data-loading and parsing workflow using D3 v5. It reads a CSV file, filters out the first three lines of metadata, and parses the remaining rows into a JavaScript array of objects. The code uses `d3.text` to fetch the file and `d3.csvParse` to convert the cleaned string into structured data. A console log then prints the parsed result, but no visualization or chart is rendered; the focus is purely on data ingestion and preprocessing. The source is a local `data.csv` file, and the approach highlights D3’s utility for handling tabular data before any graphical encoding begins.

GGerardoFurtado

56% match

Loading thumbnail…

Loading and parsing CSV data

This example demonstrates loading a remote CSV file and parsing it with D3.js, displaying basic dataset statistics as large text. It uses `d3.csv` to fetch and parse the data from a GitHub Gist, then reports the approximate size of the formatted output in kilobytes, the number of rows, and the number of columns. The page also includes commented-out code showing an alternative approach using the `fetch` API with `async`/`await` and `d3.csvParse` for manual text parsing, but the active path relies on D3's built-in CSV handling.

Aadante111

56% match

Loading thumbnail…

Loading and Parsing CSV Data - Amazon Bestsellers List Dataset

This example demonstrates loading and parsing a CSV file using D3 v5, displaying the dataset’s size in kilobytes, row count, and column count as large text. It fetches Amazon bestselling books data from a GitHub gist via `d3.csv` and processes it with `d3.csvFormat` to calculate the byte size. The visualization confirms successful data ingestion without rendering charts, focusing solely on the parsing step. The data includes book titles, authors, genres, user scores, prices, years, and review counts from 2009 to 2019.

SSitanshu Rupani

56% match

Loading thumbnail…

multilevel-http binary problem

This example demonstrates a binary data integrity test for a multilevel-http client-server setup, comparing SHA-1 hashes of content stored and retrieved through both the client and the local LevelDB database. It uses Node.js streams (fstream, tar.Pack, zlib.createGzip) to pack and compress a directory into a buffer, then stores the buffer via multilevel’s HTTP client and directly via the LevelDB `db.put` API. After retrieval, `crypto.createHash` computes and logs matching hashes, verifying binary round-trip fidelity across both access paths.

Aalessioalex

56% match

Loading thumbnail…

Gist e9eac02df35654daa5d0

This example demonstrates a minimal data pipeline that reads JSON data and inserts it into a local SQLite database. The script, `jsonToSqlite.py`, loads a JSON file containing a list of objects, extracts the first object’s `"foo"` and `"bar"` fields, and writes those two values as a single row into a table named `table_name`. It uses the standard `json` module for parsing and `sqlite3` for database operations, creating a table and inserting the data with a parameterized `execute` call. The visualization shows only the code structure, not a graphical output, highlighting the simple ETL pattern of extracting, transforming, and loading data.

CCBasis

55% match