Count disctinct values for each columns of a file with awk

100%

Count disctinct values for each columns of a file with awk

✓ Published0🌍 Public

Last edited Jul 9, 2015

Created on Jul 9, 2015

This example uses an awk script to count distinct values in every column of a CSV file, writing each column’s unique entries and their occurrence counts to a separate output CSV named after the original file and column header. The script first extracts the header row to create per-column output files, then iterates over all subsequent rows, tallying values in a two-dimensional associative array keyed by field name and value. In the END block, it splits those keys to write the final counts. The awk code relies on built-in features like the `FS` field separator, the `split` function, and the `SUBSEP` separator for array keys.

AI-generated description

Similar vizzes

Loading thumbnail…

MongoDB equivalent of an SQL query to get the distinct values of a field in a collection including the count of documents which have each distinct value (distinct with count)

This example demonstrates how to retrieve distinct values of a field in a MongoDB collection along with the document count for each value, mirroring a SQL `GROUP BY` query. It shows two aggregation pipelines using MongoDB’s `aggregate()` method with the `$group` stage to group by the field `_id` and compute a `count` via `$sum:1`, and an optional `$sort` stage to order the results by count descending. The first pipeline returns unsorted counts, while the second sorts them from highest to lowest. No external visualization library is used; the output is raw aggregated data from the database.

Aackuser

55% match

Loading thumbnail…

Linux - Count number of lines in file

This example demonstrates how to count the number of lines in a file using the `wc -l` command, displaying the result directly in the terminal. The code runs `wc -l <myfile>` and prints the output, such as `4577465 myfile.csv`, showing both the line count and the filename. It relies solely on the standard Unix `wc` utility via a shell command, with no external visualization libraries or rendering APIs. The approach is a straightforward command-line operation, not a graphical data visualization, and the output is plain text rather than a chart or interactive graphic.

WWilliamQLiu

53% match

Loading thumbnail…

MongoDB Aggregation Framework way for "select count (distinct fieldName) from someTable" for large collections

This example shows how to compute the equivalent of a SQL “select count (distinct fieldName) from someTable” using MongoDB’s Aggregation Framework for large collections. The code uses the `db.someCollection.aggregate()` method with two `$group` stages: the first groups documents by the distinct value of `fieldName` (assigned to `_id`), and the second groups all results into a single document (`_id: 1`) while summing the number of groups with `$sum: 1` to produce the final count. The snippet is written as a JavaScript gist and relies solely on MongoDB’s native aggregation pipeline, with no external visualization library.

Aackuser

51% match

Loading thumbnail…

[统计mongodb 中某个 key 对应的 value 的出现频数] 统计一个 mongodb 数据库的 collection 中，所有 documents 内，统计某个特定字段的不同值各自出现次数。 #mongodb #pymongo

This example shows how to count the frequency of distinct values for a specified field across all documents in a MongoDB collection. The code defines a Python function using PyMongo’s `aggregate` method with MongoDB aggregation pipelines. It handles both simple fields and fields containing arrays of subdocuments, where a subfield is specified. The function filters documents with `$match`, unwinds arrays with `$unwind`, groups by value with `$group`, and sorts results by count using `$sort`.

SShawn91

50% match

Loading thumbnail…

Gist 776981b42d618f2717fb986bbf501186

This example counts the number of sequences in a FASTA file, displaying the total as a plain numeric output. It reads the file line by line and increments a counter whenever a line begins with the `>` character, which marks a new sequence header. The tool is a Perl script that relies on the standard Perl file-handling and regular-expression capabilities rather than any external visualization library. The output is a single integer representing the number of contigs, scaffolds, or genes, depending on the input file's content. The approach is purely computational, with no graphical rendering involved.

NNpfn

50% match

Loading thumbnail…

Append string on each line to csv file

This example demonstrates a shell script that appends the string `"active"` to the end of every line in a CSV file, producing a new output file. It uses the `sed` command with a regular expression to match the end of each line (including lines with or without a trailing carriage return) and replaces it with `,"active"`. The script reads from `file.csv` and writes the modified content to `out.csv`. The transformation relies on POSIX `sed` in extended regex mode, paired with a Bash-specific ANSI-C quoting trick to handle the `\r` character.

Aadkdev

49% match

Loading thumbnail…

Split large file by number of lines

This example demonstrates how to split a large file into smaller, more manageable pieces by specifying a line count. It shows the command-line utility `split` dividing a source file into multiple output files, each containing exactly one million lines. The code uses the `-l` flag to set the line limit and a prefix (`data.`) to generate sequential filenames like `data.aa`, `data.ab`, and `data.ac`. The rendering approach is purely text-based, illustrating a shell command and its expected output files.

WWilliamQLiu

48% match

Loading thumbnail…

Useful csvkit commands

This example demonstrates the versatility of csvkit, a command-line toolkit for working with tabular data, by presenting a collection of common workflows. It shows how to inspect, filter, merge, and transform CSV files using commands like `csvstat`, `csvcut`, `csvlook`, and `csvgrep`. The visualization highlights chaining commands with pipes to combine operations, such as using `csvsql` for SQL queries and joins across multiple files, `csvjson` to convert data into JSON or GeoJSON, and `csvstack` to merge datasets. It also covers importing and exporting data to and from SQL databases via `csvsql` and `sql2csv`, using Postgres and MSSQL connection strings.

WWilliamQLiu

48% match