Hemodynamic effects of decompressive craniectomy in cold induced brain oedema.

A. Rinaldi,A. Mangiola,C. Anile,Giulio Maira,P. Amante,A. Ferraresi

DOI: https://doi.org/10.1007/978-3-7091-9115-6_132

Abstract:

What problem does this paper attempt to address?

Dimensionality-Aware Outlier Detection: Theoretical and Experimental Analysis

Alastair Anderberg,James Bailey,Ricardo J. G. B. Campello,Michael E. Houle,Henrique O. Marques,Miloš Radovanović,Arthur Zimek

2024-04-21

Abstract:We present a nonparametric method for outlier detection that takes full account of local variations in intrinsic dimensionality within the dataset. Using the theory of Local Intrinsic Dimensionality (LID), our 'dimensionality-aware' outlier detection method, DAO, is derived as an estimator of an asymptotic local expected density ratio involving the query point and a close neighbor drawn at random. The dimensionality-aware behavior of DAO is due to its use of local estimation of LID values in a theoretically-justified way. Through comprehensive experimentation on more than 800 synthetic and real datasets, we show that DAO significantly outperforms three popular and important benchmark outlier detection methods: Local Outlier Factor (LOF), Simplified LOF, and kNN.

Machine Learning,Artificial Intelligence
Comparison of Data Visualization, Outlier Detection and Data Dimensionality Reduction Methods

Xingyu Zhao

DOI: https://doi.org/10.54097/wgchmc87

2024-03-13

Abstract:With the deepening of the digital age of information, people's daily data is getting larger and larger, and it is more and more difficult to quantify and process. At this time, the data processing means becomes particularly important. This paper compares and analyzes some methods from data visualization to data dimensionality reduction to outlier detection. In this paper, two different types of datasets, ModelNet40, and red wine quality, are used to introduce the visualization method of the Farthest Point Sampling (FPS). This method can have a clear visual effect on the data dimension and scale, and allow users to observe the structure, type, and scale of the data. In data dimensionality reduction, the study uses Principal Component Analysis (PCA), T-Distributed Stochastic Neighbor Embedding (t-SNE), Triplets Manifold Approximation and Projection (TriMAP), Uniform Manifold Approximation and Projection (UMAP), Pairwise Controlled Manifold Approximation Projection (PaCMAP), and Autoencoder to compare their dimensionality reduction effects. Through these methods, this paper finds that different methods have different effects on different datasets. Therefore, in data dimensionality reduction, it can get twice the result with half the effort by choosing the appropriate method. Finally, this paper also detects outliers. Outliers in datasets will make it difficult for people to process data and make subsequent results inaccurate, so it is necessary to identify outliers. This paper involves methods such as isolation forest and Density-Based Spatial Clustering of Applications with Noise (DBSCAN). Through this paper, the methods of different datasets are analyzed and summarized.
Distance-based outlier detection for high dimension, low sample size data

Jeongyoun Ahn,Myung Hee Lee,Jung Ae Lee

DOI: https://doi.org/10.1080/02664763.2018.1452901

IF: 1.416

2018-03-24

Journal of Applied Statistics

Abstract:Despite the popularity of high dimension, low sample size data analysis, there has not been enough attention to the sample integrity issue, in particular, a possibility of outliers in the data. A new outlier detection procedure for data with much larger dimensionality than the sample size is presented. The proposed method is motivated by asymptotic properties of high-dimensional distance measures. Empirical studies suggest that high-dimensional outlier detection is more likely to suffer from a swamping effect rather than a masking effect, thus yields more false positives than false negatives. We compare the proposed approaches with existing methods using simulated data from various population settings. A real data example is presented with a consideration on the implication of found outliers.

statistics & probability
Detecting Outliers in High Dimensional Data Sets using Z-Score Methodology

Peruri Venkataanusha*,Ch. Anuradha,Dr. Patnala S.R. Chandra Murty,Dr. Surya Kian Chebrolu,,,,

DOI: https://doi.org/10.35940/ijitee.a3910.119119

2019-11-10

International Journal of Innovative Technology and Exploring Engineering

Abstract:Outlier detection is an interesting research area in machine learning. With the recently emergent tools and varied applications, the attention of outlier recognition is growing significantly. Recently, a significant number of outlier detection approaches have been observed and effectively applied in a wide range of fields, comprising medical health, credit card fraud and intrusion detection. They can be utilized for conservative data analysis. However, Outlier recognition aims to discover sequence in data that do not conform to estimated performance. In this paper, we presented a statistical approach called Z-score method for outlier recognition in high-dimensional data. Z-scores is a novel method for deciding distant data based on data positions on charts. The projected method is computationally fast and robust to outliers’ recognition. A comparative Analysis with extant methods is implemented with high dimensional datasets. Exploratory outcomes determines an enhanced accomplishment, efficiency and effectiveness of our projected methods.
Robust Subspace Outlier Detection in High Dimensional Space

Zhana Bao

DOI: https://doi.org/10.48550/arXiv.1405.0869

2014-05-05

Abstract:Rare data in a large-scale database are called outliers that reveal significant information in the real world. The subspace-based outlier detection is regarded as a feasible approach in very high dimensional space. However, the outliers found in subspaces are only part of the true outliers in high dimensional space, indeed. The outliers hidden in normal-clustered points are sometimes neglected in the projected dimensional subspace. In this paper, we propose a robust subspace method for detecting such inner outliers in a given dataset, which uses two dimensional-projections: detecting outliers in subspaces with local density ratio in the first projected dimensions; finding outliers by comparing neighbor's positions in the second projected dimensions. Each point's weight is calculated by summing up all related values got in the two steps projected dimensions, and then the points scoring the largest weight values are taken as outliers. By taking a series of experiments with the number of dimensions from 10 to 10000, the results show that our proposed method achieves high precision in the case of extremely high dimensional space, and works well in low dimensional space.

Artificial Intelligence,Machine Learning
Dimensionality-reduced Secure Outlier Detection on Union of Subspaces

Kunzan Liu,Yuchen Jiao,Ye Jin,Xu Xiang,Yuantao Gu

DOI: https://doi.org/10.1109/csp51677.2021.9357502

2021-01-01

Abstract:In the problem of outlier detection (OD) on a union of subspaces (UoS), inliers are assumed to lie around a union of low-dimensional subspaces, and the goal is to detect the outliers that are not close to any of these subspaces. Among various algorithms, sparse self-representation-based ones have attracted much attention because of their theoretical performance guarantee. However, these algorithms need direct access to all raw data, and thus have poor data security and privacy protection capability. To solve this problem, in this paper we propose a new algorithm called dimensionality-reduced secure outlier detection (DrSOD), which uses random projection as a preprocessing step to avoid direct access to the raw data. We theoretically prove that DrSOD can correctly detect outliers with overwhelming probability under connectivity assumptions. In addition, the random projection step improves the computational efficiency of the algorithm. Experiments on synthetic and real-world datasets also demonstrate the effectiveness and efficiency of DrSOD.
Various dimension reduction techniques for high dimensional data analysis: a review

Papia Ray,S. Surender Reddy,Tuhina Banerjee

DOI: https://doi.org/10.1007/s10462-020-09928-0

IF: 9.588

2021-01-08

Artificial Intelligence Review

Abstract:In the era of healthcare, and its related research fields, the dimensionality problem of high dimensional data is a massive challenge as it contains a huge number of variables forming complex data matrices. The demand for dimension reduction of complex data is growing immensely to improvise data prediction, analysis and visualization. In general, dimension reduction techniques are defined as a compression of dataset from higher dimensional matrix to lower dimensional matrix. Several computational techniques have been implemented for data dimension reduction, which is further segregated into two categories such as feature extraction and feature selection. In this review, a detailed investigation of various feature extraction and feature selection methods has been carried out with a systematic comparison of several dimension reduction techniques for the analysis of high dimensional data and to overcome the problem of data loss. Then, some case studies are also cited to verify the better approach for data dimension reduction by considering few advances described in the technical literature. This review paper may guide researchers to choose the most effective method for satisfactory analysis of high dimensional data.

computer science, artificial intelligence
Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series

Mahsun Altin,Altan Cakir

DOI: https://doi.org/10.1109/access.2024.3415088

IF: 3.9

2024-06-25

IEEE Access

Abstract:This paper presents an extensive empirical study on the integration of dimensionality reduction techniques with advanced unsupervised time series anomaly detection models, focusing on the MUTANT and Anomaly-Transformer models. The study involves a comprehensive evaluation across three different datasets: MSL, SMAP, and SWaT. Each dataset poses unique challenges, allowing for a robust assessment of the models' capabilities in varied contexts. The dimensionality reduction techniques examined include PCA, UMAP, Random Projection, and t-SNE, each offering distinct advantages in simplifying high-dimensional data. Our findings reveal that dimensionality reduction not only aids in reducing computational complexity but also significantly enhances anomaly detection performance in certain scenarios. Moreover, a remarkable reduction in training times was observed, with reductions by approximately 300% and 650% when dimensionality was halved and minimized to the lowest dimensions, respectively. This efficiency gain underscores the dual benefit of dimensionality reduction in both performance enhancement and operational efficiency. The MUTANT model exhibits notable adaptability, especially with UMAP reduction, while the Anomaly-Transformer demonstrates versatility across various reduction techniques. These insights provide a deeper understanding of the synergistic effects of dimensionality reduction and anomaly detection, contributing valuable perspectives to the field of time series analysis. The study underscores the importance of selecting appropriate dimensionality reduction strategies based on specific model requirements and dataset characteristics, paving the way for more efficient, accurate, and scalable solutions in anomaly detection.

computer science, information systems,telecommunications,engineering, electrical & electronic
A geometric framework for outlier detection in high-dimensional data

Moritz Herrmann,Florian Pfisterer,Fabian Scheipl

DOI: https://doi.org/10.48550/arXiv.2207.00367

2022-07-29

Abstract:Outlier or anomaly detection is an important task in data analysis. We discuss the problem from a geometrical perspective and provide a framework that exploits the metric structure of a data set. Our approach rests on the manifold assumption, i.e., that the observed, nominally high-dimensional data lie on a much lower dimensional manifold and that this intrinsic structure can be inferred with manifold learning methods. We show that exploiting this structure significantly improves the detection of outlying observations in high-dimensional data. We also suggest a novel, mathematically precise, and widely applicable distinction between distributional and structural outliers based on the geometry and topology of the data manifold that clarifies conceptual ambiguities prevalent throughout the literature. Our experiments focus on functional data as one class of structured high-dimensional data, but the framework we propose is completely general and we include image and graph data applications. Our results show that the outlier structure of high-dimensional and non-tabular data can be detected and visualized using manifold learning methods and quantified using standard outlier scoring methods applied to the manifold embedding vectors.

Machine Learning
Provable Self-Representation Based Outlier Detection in a Union of Subspaces

Chong You,Daniel P. Robinson,René Vidal

DOI: https://doi.org/10.48550/arXiv.1704.03925

2017-04-13

Abstract:Many computer vision tasks involve processing large amounts of data contaminated by outliers, which need to be detected and rejected. While outlier detection methods based on robust statistics have existed for decades, only recently have methods based on sparse and low-rank representation been developed along with guarantees of correct outlier detection when the inliers lie in one or more low-dimensional subspaces. This paper proposes a new outlier detection method that combines tools from sparse representation with random walks on a graph. By exploiting the property that data points can be expressed as sparse linear combinations of each other, we obtain an asymmetric affinity matrix among data points, which we use to construct a weighted directed graph. By defining a suitable Markov Chain from this graph, we establish a connection between inliers/outliers and essential/inessential states of the Markov chain, which allows us to detect outliers by using random walks. We provide a theoretical analysis that justifies the correctness of our method under geometric and connectivity assumptions. Experimental results on image databases demonstrate its superiority with respect to state-of-the-art sparse and low-rank outlier detection methods.

Computer Vision and Pattern Recognition,Machine Learning
A Novel Outlier Detection Method for Multivariate Data

Yahya Almardeny,Noureddine Boujnah,Frances Cleary

DOI: https://doi.org/10.1109/tkde.2020.3036524

IF: 9.235

2020-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Detecting anomalous objects from given data has a broad range of real-world applications. Although there is a rich number of outlier detection algorithms, most of them involve hidden assumptions and restrictions. This paper proposes a novel, yet effective outlier learning algorithm that is based on decomposing the full attributes space into different combinations of subspaces, in which the 3D-vectors, representing the data points per 3D-subspace, are rotated about the geometric median, using Rodrigues rotation formula, to construct the overall outlying score. The proposed approach is parameter-free, requires no distribution assumptions and easy to implement. Extensive experimental study and comparison are conducted on both synthetic and real-world datasets with six popular outlier detection algorithms, each from different category. The comparison is evaluated based on the precision @s, average precision, rank power, AUC ROC and time complexity metrics. The results show that the performance of the proposed method is competitive and promising.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Improved Method for Noise Detection by DBSCAN and Angle Based Outlier Factor in High Dimensional Datasets

Sarita Tripathy,Laxman Sahoo

DOI: https://doi.org/10.1007/978-981-13-8715-9_27

2019-08-02

Abstract:Various data mining methods are used to detect outliers from different databases. It is essential to detect outliers in different kinds of real time application areas, such as the health care sector and insurance, marketing, banks and finance et c. The proposed method in this paper is a new outlier detection method from a high dimensional data sets which combines the angle based outlier detection (ABOD) with the classical density based clustering method DBSCAN. The algorithm consists of three stages in which the first stage consists of applying of the PCA on the data set which will result in a subset of attributes, to this subset of attributes the dbscan algorithm is applied which results in detection of a set of outliers. In the third and final stage ABOD is applied to the set of outliers. Experimental analysis conducted state that the result improves the detection accuracy and decreases the number of false positives.
Outlier Detection and Spatial Analysis Algorithms

Jacob John

DOI: https://doi.org/10.48550/arXiv.2106.10669

2021-06-20

Abstract:Outlier detection is a significant area in data mining. It can be either used to pre-process the data prior to an analysis or post the processing phase (before visualization) depending on the effectiveness of the outlier and its importance. Outlier detection extends to several fields such as detection of credit card fraud, network intrusions, machine failure prediction, potential terrorist attacks, and so on. Outliers are those data points with characteristics considerably different. They deviate from the data set causing inconsistencies, noise and anomalies during analysis and result in modification of the original points However, a common misconception is that outliers have to be immediately eliminated or replaced from the data set. Such points could be considered useful if analyzed separately as they could be obtained from a separate mechanism entirely making it important to the research question. This study surveys the different methods of outlier detection for spatial analysis. Spatial data or geospatial data are those that exhibit geographic properties or attributes such as position or areas. An example would be weather data such as precipitation, temperature, wind velocity, and so on collected for a defined region.

Machine Learning
A robust distance-based approach for detecting multidimensional outliers

R. Lakshmi,T. A. Sajesh

DOI: https://doi.org/10.1080/02664763.2024.2422403

IF: 1.416

2024-11-09

Journal of Applied Statistics

Abstract:Identifying outliers in data analysis is a critical task, as outliers can significantly influence the results and conclusions drawn from a dataset. This study explores the use of the Mahalanobis distance metric for detecting outliers in multivariate data, focusing on a novel approach inspired by the work of M. Falk, [ On mad and comedians , Ann. Inst. Stat. Math. 49 (1997), pp. 615–644]. The proposed method is rigorously tested through extensive simulation analysis, where it demonstrates high True Positive Rates (TPR) and low False Positive Rates (FPR) when compared to other existing outlier detection techniques. Through extensive simulation analysis, we empirically evaluate the affine equivariance and breakdown properties of our proposed distance measure and it is evident from the outputs that our robust distance measure demonstrates effective results with respect to the measures FPR and TPR. The proposed method was applied to seven different datasets, showing promising true positive rates (TPR) and false positive rates (FPR), and it outperformed several well-known outlier identification approaches. We can effectively use our proposed distance measure in fields demanding outlier detection.

statistics & probability
Modern Dimension Reduction

Philip D. Waggoner

DOI: https://doi.org/10.48550/arXiv.2103.06885

2021-03-11

Abstract:Data are not only ubiquitous in society, but are increasingly complex both in size and dimensionality. Dimension reduction offers researchers and scholars the ability to make such complex, high dimensional data spaces simpler and more manageable. This Element offers readers a suite of modern unsupervised dimension reduction techniques along with hundreds of lines of R code, to efficiently represent the original high dimensional data space in a simplified, lower dimensional subspace. Launching from the earliest dimension reduction technique principal components analysis and using real social science data, I introduce and walk readers through application of the following techniques: locally linear embedding, t-distributed stochastic neighbor embedding (t-SNE), uniform manifold approximation and projection, self-organizing maps, and deep autoencoders. The result is a well-stocked toolbox of unsupervised algorithms for tackling the complexities of high dimensional data so common in modern society. All code is publicly accessible on Github.

Machine Learning,Computers and Society,Applications
Real-valued Multivariate Dimension Reduction: A Survey

Hong-Ming SHAN,Jun-Ping ZHANG

DOI: https://doi.org/10.16383/j.aas.2018.c160812

2018-01-01

Abstract:As one of the classical problems in machine learning,dimension reduction is used for dealing with the curse of dimensionality,speeding up computational efficiency of the algorithm,and improving interpretability as well as visualizing high-dimensional data.Traditional dimension reduction algorithms such as principal component analysis (PCA) and linear discriminant analysis are mainly suitable for unlabeled data or classification data.When the response variables are univariate or multivariate continuous real-valued ones,however,such dimension reduction methods cannot guarantee the effective predictive performance of the reduced subspace.In the recent two decades,researchers have been devoted to studying this issue with different viewpoints,attaining many promising and systemic achievements.Under this background,we will survey the developments of real-valued multivariate dimension reduction in detail.We will also introduce its basic concepts,algorithms and theories,and discuss some potential research directions deserving investigating.
Feature dimensionality reduction: a review

Weikuan Jia,Meili Sun,Jian Lian,Sujuan Hou

DOI: https://doi.org/10.1007/s40747-021-00637-x

2022-01-21

Abstract:Abstract As basic research, it has also received increasing attention from people that the “curse of dimensionality” will lead to increase the cost of data storage and computing; it also influences the efficiency and accuracy of dealing with problems. Feature dimensionality reduction as a key link in the process of pattern recognition has become one hot and difficulty spot in the field of pattern recognition, machine learning and data mining. It is one of the most challenging research fields, which has been favored by most of the scholars’ attention. How to implement “low loss” in the process of feature dimension reduction, keep the nature of the original data, find out the best mapping and get the optimal low dimensional data are the keys aims of the research. In this paper, two-dimensionality reduction methods, feature selection and feature extraction, are introduced; the current mainstream dimensionality reduction algorithms are analyzed, including the method for small sample and method based on deep learning. For each algorithm, examples of their application are given and the advantages and disadvantages of these methods are evaluated.

computer science, artificial intelligence
Overview and comparative study of dimensionality reduction techniques for high dimensional data

Shaeela Ayesha,Muhammad Kashif Hanif,Ramzan Talib

DOI: https://doi.org/10.1016/j.inffus.2020.01.005

IF: 18.6

2020-07-01

Information Fusion

Abstract:<p>The recent developments in the modern data collection tools, techniques, and storage capabilities are leading towards huge volume of data. The dimensions of data indicate the number of features that have been measured for each observation. It has become a challenging task to analyze high dimensional data. Different dimensionality reduction techniques are available in literature to eliminate irrelevant and redundant features. Selection of an appropriate dimension reduction technique can help to enhance the processing speed and reduce the time and effort required to extract valuable information. This paper presents the state-of-the art dimensionality reduction techniques and their suitability for different types of data and application areas. Furthermore, the issues of dimensionality reduction techniques have been highlighted that can affect the accuracy and relevance of results.</p>

computer science, artificial intelligence, theory & methods
Statistical Inference from High Dimensional Data

Xia Yin

DOI: https://doi.org/10.3390/books978-3-0365-0945-7

2021-01-01

Abstract:• Real-world problems can be high-dimensional, complex, and noisy • More data does not imply more information • Different approaches deal with the so-called curse of dimensionality to reduce irrelevant information • A process with multidimensional information is not necessarily easy to interpret nor process • In some real-world applications, the number of elements of a class is clearly lower than the other. The models tend to assume that the importance of the analysis belongs to the majority class and this is not usually the truth • The analysis of complex diseases such as cancer are focused on more-than-one dimensional omic data • The increasing amount of data thanks to the reduction of cost of the high-throughput experiments opens up a new era for integrative data-driven approaches • Entropy-based approaches are of interest to reduce the dimensionality of high-dimensional data

Hemodynamic effects of decompressive craniectomy in cold induced brain oedema.

Dimensionality-Aware Outlier Detection: Theoretical and Experimental Analysis

Comparison of Data Visualization, Outlier Detection and Data Dimensionality Reduction Methods

Distance-based outlier detection for high dimension, low sample size data

Detecting Outliers in High Dimensional Data Sets using Z-Score Methodology

Robust Subspace Outlier Detection in High Dimensional Space

Dimensionality-reduced Secure Outlier Detection on Union of Subspaces

Various dimension reduction techniques for high dimensional data analysis: a review

Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series

A geometric framework for outlier detection in high-dimensional data

Provable Self-Representation Based Outlier Detection in a Union of Subspaces

A Novel Outlier Detection Method for Multivariate Data

Improved Method for Noise Detection by DBSCAN and Angle Based Outlier Factor in High Dimensional Datasets

Outlier Detection and Spatial Analysis Algorithms

A robust distance-based approach for detecting multidimensional outliers

Modern Dimension Reduction

Real-valued Multivariate Dimension Reduction: A Survey

Feature dimensionality reduction: a review

Overview and comparative study of dimensionality reduction techniques for high dimensional data

Statistical Inference from High Dimensional Data