Abstract:Current generation of Internet-based services are typically hosted on large data centers that take the form of warehouse-size structures housing tens of thousands of servers. Continued availability of a modern data center is the result of a complex orchestration among many internal and external actors including computing hardware, multiple layers of intricate software, networking and storage devices, electrical power and cooling plants. During the course of their operation, many of these components produce large amounts of data in the form of event and error logs that are essential not only for identifying and resolving problems but also for improving data center efficiency and management. Most of these activities would benefit significantly from data analytics techniques to exploit hidden statistical patterns and correlations that may be present in the data. The sheer volume of data to be analyzed makes uncovering these correlations and patterns a challenging task. This paper presents Big Data analyzer (BiDAl), a prototype Java tool for log-data analysis that incorporates several Big Data technologies in order to simplify the task of extracting information from data traces produced by large clusters and server farms. BiDAl provides the user with several analysis languages (SQL, R and Hadoop MapReduce) and storage backends (HDFS and SQLite) that can be freely mixed and matched so that a custom tool for a specific task can be easily constructed. BiDAl has a modular architecture so that it can be extended with other backends and analysis languages in the future. In this paper we present the design of BiDAl and describe our experience using it to analyze publicly-available traces from Google data clusters, with the goal of building a realistic model of a complex data center.

On the Verge of One Petabyte - the Story Behind the BaBar Database System

Improving the Security and Performance of the BaBar Detector Controls System

BaBar Web job submission with Globus authentication and AFS access

Objectivity Data Migration

The BaBar Electromagnetic Calorimeter: Status and Performance Improvements

InfiniteDB: a pc-cluster based parallel massive database management system.

Blue Gene: A vision for protein science using a petaflop supercomputer

Measuring the Contributions of (O)RDBMS to Object-Oriented Software Development

Benchmarking SciDB Data Import on HPC Systems

Eventdb: A Large-Scale Semi-Structured Scientific Data Management System

AZDBLab

Offline Data Processing System of the BESIII Experiment

Statistical Issues in Particle Physics -- A View from BaBar

Scalable Database Access Technologies for ATLAS Distributed Computing

The MultiDark Database: Release of the Bolshoi and MultiDark Cosmological Simulations

Toward real-time data query systems in HEP

A High Energy Nuclear Database Proposal

H-DB: Yet Another Big Data Hybrid System of Hadoop and DBMS

Banian: A Cross-Platform Interactive Query System for Structured Big Data

A Big Data analyzer for large trace logs

Data Acquisition Software for CBM-TOF Super Module Quality Control