Skip to main content
100%

Apache log analysis with Hadoop, Hive and HBase

✓ Published0🌍 Public
Aadkdev
Last edited Jan 3, 2012
Created on May 19, 2016

This example demonstrates how to analyze Apache web server logs using Hadoop, Hive, and HBase. It shows the process of transforming raw gzipped log files stored on the Hadoop Distributed File System into a structured HBase table, then querying that table to find the 50 most popular pages and categorize article hits by publication year. The Hive script uses the RegexSerDe to parse log lines, the HBaseStorageHandler to map Hive columns to HBase column families (m, r, a), and SQL-like queries with functions such as `unix_timestamp`, `re_extract`, and `TABLESAMPLE` to clean and aggregate the data.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist 9ea11dbcc52b4bb1c048

This example shows performance analysis for a data engineering capstone project, comparing a legacy Python log parser against a proposed Scala implementation on Spark and Mesos. The visualization tracks transfer metrics from ASCP log data, specifically examining throughput and packet rates. Using d3.js for charting, it evaluates how efficiently the new streaming solution handles regex parsing of unstructured logs. The project also demonstrates integration with Redis for persistent storage and FaspStream for live data ingestion, with the D3 visualizations presenting benchmark results from test log datasets.

BBBischof
61% match
Loading thumbnail…

Gist 38a5a981b671bf09a7b9e76eeebc2294

This example visualizes a system-administration script that automates the setup of an Ambari-managed Hadoop cluster. The code shows a sequence of shell commands and configuration steps, including disabling SELinux and iptables, installing Java and MySQL, and configuring databases and users. It uses the `awk` utility to edit `sshd_config` and the `mysql` command-line client to create and grant privileges on the `ambaridb`, `hivedb`, and `ooziedb` databases. The visualization maps the script’s control flow and dependencies, likely through a node-link diagram, to illustrate how each step builds on prior ones.

FFred-Fan
58% match
Loading thumbnail…

Gist f6017501c4c3249dbb32

This example parses a data-transfer log line from Aspera’s ASCP protocol, converting its space-delimited key/value pairs into a structured JSON object. It demonstrates a pure Scala regex-based parser, processing each field (bandwidth, retransmission, delay) for downstream analysis. The code uses Scala’s `scala.util.matching.Regex` and `String` methods to extract and format the metrics, serving as a first step toward a streaming Spark pipeline.

BBBischof
55% match
Loading thumbnail…

Gist 25342708e9edeb5099b496447772d743

The visualization shows a technique for analyzing log files by measuring time gaps between consecutive timestamped entries. It uses Python’s `datetime.strptime` to parse timestamps from `"YYYY-MM-DD HH:MM:SS.mmm"` formats and calculates differences in milliseconds, flagging intervals exceeding 10 ms. The code filters log lines by keyword using standard file I/O operations and prints matching entries, or reads all lines from a file to identify timing anomalies. This example demonstrates text-based log inspection rather than graphical rendering, relying on Python console output for insights.

JJastor11
54% match
Loading thumbnail…

Clear Hadoop tmp

This visualization shows the file cleanup process for the Hadoop temporary directory, displaying a chronological flow of deleted files. It renders a tree-like structure of the `/opt/hadoop/tmp` path, with each node representing a file or folder, and animates their removal as the shell script executes. The code uses a Bash script with `find` to locate files older than 360 minutes and `rm` to delete them, while the visualization likely relies on a custom layout to map the directory hierarchy. Data is drawn directly from the script’s output, showing which entries expire over time.

Aadkdev
53% match
Loading thumbnail…

Gist d2f2bc6be1716820160ed5183ea07eaa

This example shows a browser-based text analytics pipeline that scrapes and processes web content before visualizing topic models. The code first uses Selenium with Chrome WebDriver to scrape data, then applies NLP libraries including NLTK, spaCy, TextBlob, and gensim’s LdaModel for topic modeling and coherence scoring. The final visualization employs Plotly’s `plotly.express` and `plotly.graph_objs` to render interactive charts, with `pyo.init_notebook_mode()` enabling notebook display. The rendering approach is evident from the use of `go` traces and `px` figures, though the specific chart type is not shown in the code snippet. The data source is dynamic web pages, as indicated by the Selenium scraping logic.

AArizonaTay
53% match
Loading thumbnail…

1. read.tsv, 2. lodash processing, 3. Apache ECharts

This example shows a bar chart of student counts by academic program, generated from a small inline dataset of repeated degree names. The code first reads a TSV file conceptually, then uses Lodash’s `_.countBy` to aggregate the raw array by the `c` property, producing a frequency object. The resulting counts are logged to the console and then visualized using Apache ECharts, with the chart container defined as a `div` element. The data is placeholder values, not real file input, and the ECharts configuration is left incomplete.

Aaaizemberg
52% match
Loading thumbnail…

Gist ed5d1681970062ebe8088ea3fdbb896f

This example shows a live query log for a Power BI or Analysis Services session by displaying the most recent command timestamps and command texts. It starts with an empty table, loads prior log rows, then queries the current session’s last command data using the `MetaQuery` and `SELECT` statements against the `$SYSTEM.DISCOVER_SESSIONS` schema. The code filters out its own log entries, compares timestamps to keep only new rows since the last refresh, and appends them to the existing log. The rendering relies on Power Query’s table operations (`Table.FromRows`, `Table.SelectRows`, `Table.Combine`) rather than a visual JavaScript library.

HHugoberry
52% match