ndjson-to-file

100%

ndjson-to-file

✓ Published0🌍 Public

Last edited Jan 21, 2017

Created on Jan 21, 2017

This example explores the challenge of writing large JSON datasets to disk. It demonstrates a failing workaround that serializes data in chunks to multiple files, and two broken attempts at using streaming libraries. The code uses `JSON.stringify` with `fs.writeFileSync` for the chunked approach, and attempts to use the `ndjson` and `JSONStream` libraries with `fs.createWriteStream` and piping for streaming output. The example highlights the difficulty of handling big data blobs that crash `JSON.stringify` and the struggle to implement a proper streaming solution.

AI-generated description

Trying to save a big blob of JSON that crashes JSON.stringify. Spliting into smaller arrays and saving works, but I can't figure out how to use a stream.

Similar vizzes

Loading thumbnail…

fresh block

This example demonstrates how to load multiple JSON data files sequentially and log their contents to the browser console. It shows three separate data files, each containing an object with `foo` and `bar` properties, and uses `Promise.all` with an array of `d3.json` requests to fetch all files concurrently. The code relies on D3 v5’s promise-based asynchronous loading to handle the file requests, then logs the resolved array of parsed objects. No visualization is rendered; the focus is purely on data loading and promise handling.

GGerardoFurtado

55% match

Loading thumbnail…

Read in JSON department files

This example demonstrates how to load and integrate data from multiple JSON files using D3.js v5. It reads two datasets—department names and course pathways—via `Promise.all` and `d3.json`, then logs both to the console. The visualization renders a list of course names as plain text paragraphs, appended to the body with `d3.select("body").selectAll("p")` and `.data(pathwaysData)`. The code highlights asynchronous data loading and simple data binding without any graphical marks or scales.

NNourAldinAlmubarak

55% match

Loading thumbnail…

fresh block

This example demonstrates loading multiple JSON data files concurrently using D3 v5. It shows how to fetch three separate datasets and combine their results into a single array for further processing. The code uses `d3.json()` to request each file and `Promise.all()` to wait for all requests to complete before logging the combined values to the browser console. The visualization itself is not rendered—instead, the example focuses on the data-loading pattern, with the three JSON files containing simple objects with `foo` and `bar` properties. The approach highlights asynchronous data handling in D3 without any chart or graphical output.

GGerardoFurtado

55% match

Loading thumbnail…

Gist 597373

This example shows a Node.js script that attempts to process crime incident data from a CSV file using the `ecsv` library, but it fails due to a system resource limit. The code calls `exec('wc -c "foo"', ...)` inside the `ecsv.each` callback for every row, spawning a new child process for each record. The visible stack trace from `child_process.spawn` reveals the error "Too many open files," indicating the script exhausts file descriptors because it launches processes asynchronously without waiting for completion. The gist demonstrates a common anti-pattern where unthrottled asynchronous process spawning overwhelms the operating system, using only Node.js core APIs (`child_process.exec`, `fs`) and the third-party `ecsv` parser.

CCaged

55% match

Loading thumbnail…

Gist f0b5295ac06573fd875dac15dc6fe1e1

This example logs a shuffled array and fetches JSON data from a Dallas News data store containing Cowboys draft history. Using jQuery for DOM readiness and Lodash's `shuffle`, it imports a custom `getdata` module that relies on `d3-fetch`'s `json` function to asynchronously retrieve the dataset and output it to the console. The code also demonstrates commented-out alternatives with jQuery's `$.getJSON`, showing a work-in-progress approach to loading and logging the same external JSON resource.

LLayne Smith

54% match

Loading thumbnail…

Gist 66b1a9a31dce76682837c3fd4d9a468b

This example demonstrates how to query JSONB data directly in PostgreSQL and flatten it into a relational result set. The code creates a table with a JSONB column, inserts two sample rows, and then uses two different query approaches: the `->>` operator to extract scalar fields as text, and the `jsonb_to_record` function with a column definition list to cast JSON values into typed columns (text and integer). The example highlights both methods for accessing nested JSON data, showing the explicit extraction syntax versus the more structured record-expansion technique, all within a single SQL file.

JJonasHeylen

54% match

Loading thumbnail…

fresh block

This example demonstrates how to load and process multiple JSON data files concurrently in a single D3.js application. The visualization itself is not yet rendered; instead, the code focuses on fetching `data1.json`, `data2.json`, and `data3.json` using the `d3.json` method and combining their resolved values with `Promise.all`. The resulting array of parsed JSON objects is logged to the browser console, showing the structure of the data for future use. The code relies on the D3 v5 library for its promise-based asynchronous loading, setting up a foundation for subsequent data binding or chart creation.

GGerardoFurtado

54% match

Loading thumbnail…

Bank Failure Dataset2

This example displays a raw JSON dump of bank failure data, including asset values, deposit values, and losses, loaded from a CSV file. The visualization initially shows nothing until the data is fetched, parsed with `d3.csvParse`, and rendered as a text representation using `JSON.stringify`. The code uses D3 v7’s `select` and `selectAll` to manage the display, and relies on `fetch` to retrieve the dataset from a GitHub gist. The interaction is minimal, focusing on loading and presenting the structured data.

Jjjlaber

53% match