Abstract:Current Cloud clusters often consist of heterogeneous machine nodes, which can trigger performance challenges such as the task straggler problem, whereby a small subset of parallel tasks running abnormally slower than the other sibling ones. The straggler problem leads to extended job response and deteriorates system throughput. Poor performance nodes are more likely to engender stragglers, and can undermine straggler mitigation effectiveness. For example, as the dominant mechanism for straggler alleviation, speculative execution functions by creating redundant task replicas on other machine nodes as soon as a straggler is detected. When speculative copies are assigned onto the poor performance nodes, it is hard for them to catch up with the stragglers compared to replicas run on fast nodes. And due to the fact that the performance heterogeneity is caused not only by static attribute variations such as physical capacity, but also dynamic characteristic uctuations such as contention level, analyzing node performance is important yet challenging. In this paper we develop ML-NA, a Machine Learning based Node performance Analyzer. By leveraging historical parallel tasks execution log data, ML-NA classies cluster nodes into different categories and predicts their performance in the near future as a scheduling guide to improve speculation effectiveness and minimize task straggler generation. We consider MapReduce as a representative framework to perform our analysis, and use the published OpenCloud trace as a case study to train and to evaluate our model. Results show that ML-NA can predict node performance categories with an average accuracy up to 92.86%.

Invader Job: A Kind Of Malicious Failure Job On Hadoop Yarn

A Mixed Intrusion Detection System Utilizing K-means and Extreme Gradient Boosting

Adaptive Failure Detection Via Heartbeat under Hadoop.

A Novel Intrusion Detection System Based on an Optimal Hybrid Kernel Extreme Learning Machine

SEINA: A Stealthy and Effective Internal Attack in Hadoop Systems

Predicting Misconfiguration-Induced Unsuccessful Executions of Jobs in Big Data System

Eavesdropper: A Framework for Detecting the Location of the Processed Result in Hadoop.

SEYARN: Enhancing Security of YARN Clusters Based on Health Check Service

Call Trace and Memory Access Pattern based Runtime Insider Threat Detection for Big Data Platforms

Yadoop: an Elastic Resource Management Solution of YARN.

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

A Novel Hadoop Security Model for Addressing Malicious Collusive Workers

Research on the Framework and Resource Scheduling Mechanisms of Hadoop YARN

Hunting Killer Tasks for Cloud System Through Machine Learning: A Google Cluster Case Study

ML-NA: A Machine Learning Based Node Performance Analyzer Utilizing Straggler Statistics

Analysis of Frequently Failing Tasks and Rescheduling Strategy in the Cloud System

Online Job Failure Prediction in an HPC System

Auditing of hadoop log file for dynamic detection of threats using H-ISSM-MIM and convolutional neural network

A First Step Toward Detecting Ssh Identity Theft In Hpc Cluster Environments: Discriminating Masqueraders Based On Command Behavior

Bioinformatics Computational Cluster Batch Task Profiling with Machine Learning for Failure Prediction

Time Series Based Killer Task Online Recognition Service: A Google Cluster Case Study