Abstract:The quality of the data being analyzed is a critical factor that affects the accuracy of data mining algorithms. There are two important aspects of the data quality, one is relevance and the other is data redundancy. The inclusion of irrelevant and redundant features in the data mining model results in poor predictions and high computational overhead. This paper presents an efficient method concerning both the relevance of the features and the pairwise features correlation in order to improve the prediction and accuracy of our data mining algorithm. We introduce a new feature correlation metric Q/sub Y/(X/sub i/,X/sub j/) and feature subset merit measure e(S) to quantify the relevance and the correlation among features with respect to a desired data mining task (e.g., detection of an abnormal behavior in a network service due to network attacks). Our approach takes into consideration not only the dependency among the features, but also their dependency with respect to a given data mining task. Our analysis shows that the correlation relationship among features depends on the decision task and, thus, they display different behaviors as we change the decision task. We applied our data mining approach to network security and validated it using the DARPA KDD99 benchmark data set. Our results show that, using the new decision dependent correlation metric, we can efficiently detect rare network attacks such as User to Root (U2R) and Remote to Local (R2L) attacks. The best reported detection rates for U2R and R2L on the KDD99 data sets were 13.2 percent and 8.4 percent with 0.5 percent false alarm, respectively. For U2R attacks, our approach can achieve a 92.5 percent detection rate with a false alarm of 0.7587 percent. For R2L attacks, our approach can achieve a 92.47 percent detection rate with a false alarm of 8.35 percent.

Relative Synergy Coefficient: A Novel Way to Detect Variable Interaction in Large Dataset

Feature Selection with Conditional Mutual Information Considering Feature Interaction

Detecting Beneficial Feature Interactions for Recommender Systems

Can Synergy in Triple-Helix Relations be Quantified? A Review of the Development of the Triple-Helix Indicator

An Interaction-Enhanced Feature Selection Algorithm

A new dependency and correlation analysis for features

A Novel Group-Sparsity-Optimization-Based Feature Selection Model For Complex Interaction Recognition

Interaction-based Feature Selection Using Factorial Design

High-order interaction feature selection for classification learning: A robust knowledge metric perspective

A Novel Hybrid Feature Importance and Feature Interaction Detection Framework for Predictive Optimization in Industry 4.0 Applications

Distributing Synergy Functions: Unifying Game-Theoretic Interaction Methods for Machine-Learning Explainability

Beyond Positive Similarity Metrics: Leveraging Negative Co-Occurrence in Recommender Systems

Nonparametric statistics inspired similarity measure for accuracy in collaborative filtering

Exploring the cloud of feature interaction scores in a Rashomon set

MFSynDCP: multi-source feature collaborative interactive learning for drug combination synergy prediction

A feature-based information-theoretic approach for detecting interpretable, long-timescale pairwise interactions from time series

Unifying Pairwise Interactions in Complex Dynamics

New improvements in the use of dependence measures for sensitivity analysis and screening

Error-controlled non-additive interaction discovery in machine learning models

Quantifying User Coherence: A Unified Framework for Cross-Domain Recommendation Analysis

VQSynery: Robust Drug Synergy Prediction With Vector Quantization Mechanism