Gist f6017501c4c3249dbb32

100%

Gist f6017501c4c3249dbb32

✓ Published0🌍 Public

Last edited Dec 24, 2015

Created on Dec 24, 2015

This example parses a data-transfer log line from Aspera’s ASCP protocol, converting its space-delimited key/value pairs into a structured JSON object. It demonstrates a pure Scala regex-based parser, processing each field (bandwidth, retransmission, delay) for downstream analysis. The code uses Scala’s `scala.util.matching.Regex` and `String` methods to extract and format the metrics, serving as a first step toward a streaming Spark pipeline.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist 9ea11dbcc52b4bb1c048

This example shows performance analysis for a data engineering capstone project, comparing a legacy Python log parser against a proposed Scala implementation on Spark and Mesos. The visualization tracks transfer metrics from ASCP log data, specifically examining throughput and packet rates. Using d3.js for charting, it evaluates how efficiently the new streaming solution handles regex parsing of unstructured logs. The project also demonstrates integration with Redis for persistent storage and FaspStream for live data ingestion, with the D3 visualizations presenting benchmark results from test log datasets.

BBBischof

74% match

Loading thumbnail…

Gist 25342708e9edeb5099b496447772d743

The visualization shows a technique for analyzing log files by measuring time gaps between consecutive timestamped entries. It uses Python’s `datetime.strptime` to parse timestamps from `"YYYY-MM-DD HH:MM:SS.mmm"` formats and calculates differences in milliseconds, flagging intervals exceeding 10 ms. The code filters log lines by keyword using standard file I/O operations and prints matching entries, or reads all lines from a file to identify timing anomalies. This example demonstrates text-based log inspection rather than graphical rendering, relying on Python console output for insights.

JJastor11

54% match

Loading thumbnail…

Gist 4905f6e091305c6a3ff3c1fe2c0a50b2

This example visualizes the structure of a DAX query plan by parsing a text-based log file into a hierarchical tree. It uses the Moo lexer to tokenize lines, identifiers, column references, and operator types, then builds a nested representation of the plan. The visualization likely renders this tree to show the flow of logical and physical operations, with each node displaying its operator type and associated properties such as filters, joins, or spooling details. The code defines custom token patterns for DAX-specific syntax, including column references like `'Table'[Column]` and measure references like `[Measure]`, and parses them into structured objects for display.

HHugoberry

53% match

Loading thumbnail…

CVS to JSONL example

This example shows a practical data-transformation workflow that converts a CSV file into JSONL format, producing one JSON object per line. It fetches a remote CSV file from a GitHub gist using the `fetch:text` function, parses it into records with `csv:parse`, and iterates through each record to build key-value pairs. The code distinguishes between integer and string values using `castable as xs:integer`, then constructs JSON maps with `xml-to-json` using XQuery 3.1. The output is unindented JSONL, suitable for streaming or line-oriented processing.

CCliffordAnderson

52% match

Loading thumbnail…

Spark Notes

This example shows a text-analytics pipeline that annotates a CSV-loaded corpus of academic articles. It renames raw columns to meaningful fields such as `article`, `journal`, `date`, and `text`, then uses Spark NLP’s `PretrainedPipeline("explain_document_ml")` to transform the data, adding tokens and other linguistic annotations. The code demonstrates schema inspection via `printSchema` and previews the resulting token column with `select("token").show()`, highlighting how Spark DataFrames integrate with a pretrained NLP model for scalable document processing.

CCliffordAnderson

52% match

Loading thumbnail…

Gist 86a2162840809aa2b4e4890ec2ba7100

This example demonstrates how to create and validate a JSON Web Token (JWT) using the Scala library `pdi.jwt` with ES256 asymmetric encryption. It shows the complete token lifecycle: generating a key pair with `GeneratorKeyFactory` on the `PRIME256V1` curve, encoding claims (including custom fields like `role` and `env`), and finally decoding and verifying the token with the public key. The code serializes claims using `org.json4s` and `jackson.compactJson`, highlighting the interplay between JWT construction, cryptographic signing, and verification.

CCarlos-Augusto

52% match

Loading thumbnail…

Loding and Parsing My Gas Record

This example demonstrates how to load and parse a manually recorded CSV file containing gas filling records since 2019. It displays a summary message showing the total number of rows, columns, and file size in kilobytes. The visualization uses d3.csv to fetch and parse the data from a GitHub Gist, then computes and renders the results as large text in an HTML pre element. The code highlights the simplicity of loading external tabular data and assessing its structure before further analysis.

Rrohame

52% match

Loading thumbnail…

Dataset #2 Network Flows

This example parses a truncated network traffic dataset from Stratosphere Labs, displaying a summary of its dimensions and contents. The page shows the total row count, column count, and file size in kilobytes, then prints the first ten rows as formatted JSON. The code uses D3 v5’s `csvFormat` to calculate data size and `tsv` to load the tab-separated data file. A black terminal-style interface with green monospace text echoes the dataset’s raw structure.

Ffloresh

52% match