Appropriateness of drugs prescribed by primary care physicians for depressed outpatients.

E. Gullick,L. King

DOI: https://doi.org/10.1016/0165-0327(79)90024-7

IF: 6.533

1979-03-01

Journal of Affective Disorders

Abstract:

What problem does this paper attempt to address?

Visualisation and outlier detection for probability density function ensembles

Alexander C. Murph,Justin D. Strait,Kelly R. Moran,Jeffrey D. Hyman,Philip H. Stauffer

DOI: https://doi.org/10.1002/sta4.662

2024-03-13

Stat

Abstract:Exploratory data analysis (EDA) for functional data—data objects where observations are entire functions—is a difficult problem that has seen significant attention in recent literature. This surge in interest is motivated by the ubiquitous nature of functional data, which are prevalent in applications across fields such as meteorology, biology, medicine and engineering. Empirical probability density functions (PDFs) can be viewed as constrained functional data objects that must integrate to one and be nonnegative. They show up in contexts such as yearly income distributions, zooplankton size structure in oceanography and in connectivity patterns in the brain, among others. While PDF data are certainly common in modern research, little attention has been given to EDA specifically for PDFs. In this paper, we extend several methods for EDA on functional data for PDFs and compare them on simulated data that exhibit different types of variation, designed to mimic that seen in real‐world applications. We then use our new methods to perform EDA on the breakthrough curves observed in gas transport simulations for underground fracture networks.

statistics & probability
Towards High-Performance Exploratory Data Analysis (EDA) Via Stable Equilibrium Point

Yuxuan Song,Yongyu Wang

DOI: https://doi.org/10.48550/arXiv.2306.04425

IF: 5.414

2023-06-07

Machine Learning

Abstract:Exploratory data analysis (EDA) is a vital procedure for data science projects. In this work, we introduce a stable equilibrium point (SEP) - based framework for improving the efficiency and solution quality of EDA. By exploiting the SEPs to be the representative points, our approach aims to generate high-quality clustering and data visualization for large-scale data sets. A very unique property of the proposed method is that the SEPs will directly encode the clustering properties of data sets. Compared with prior state-of-the-art clustering and data visualization methods, the proposed methods allow substantially improving computing efficiency and solution quality for large-scale data analysis tasks.
MetaInsight: Automatic Discovery of Structured Knowledge for Exploratory Data Analysis

Pingchuan Ma,Rui Ding,Shi Han,Dongmei Zhang

DOI: https://doi.org/10.1145/3448016.3457267

2021-06-09

Abstract:Automatic Exploratory Data Analysis (EDA) focuses on automatically discovering pieces of knowledge in the form of interesting data patterns. However, the conveyed knowledge by these suggested data patterns are disjointed or lack organization. Therefore, it is difficult for users to gain structured knowledge, and as the number of suggested patterns grows, these stand-alone patterns are less likely to motive users to conduct follow-up analysis, which hinders it from being effectively utilized to facilitate EDA. In this paper, we propose MetaInsight, a structured representation of knowledge extracted from multi-dimensional data aiming to facilitate EDA automatically and effectively. Specifically, we propose a novel formulation of basic data pattern to capture essential characteristics of raw data distribution to achieve knowledge extraction. Then based on the mined Homogeneous Data Patterns (HDP) and inter-pattern similarity, MetaInsight is identified by categorizing basic data patterns (within an HDP) into commonness(es) and exceptions thus achieving structured knowledge representation. The commonness(es) and exceptions concretize the knowledge obtained by induction and validation processes which are two typical analysis mechanisms conducted in EDA. We propose a novel scoring function to quantify the usefulness of MetaInsight, an effective and efficient mining procedure and a ranking algorithm to automatically discover high-quality MetaInsights from multi-dimensional data. We demonstrate the effectiveness and efficiency of MetaInsights (w.r.t. facilitating EDA) through evaluation on real-world datasets and user studies on both expert users and non-expert users.
Big data dimensional analysis

Vijay Gadepally,Jeremy Kepner

DOI: https://doi.org/10.1109/hpec.2014.7040944

2014-09-01

Abstract:The ability to collect and analyze large amounts of data is a growing problem within the scientific community. The growing gap between data and users calls for innovative tools that address the challenges faced by big data volume, velocity and variety. One of the main challenges associated with big data variety is automatically understanding the underlying structures and patterns of the data. Such an understanding is required as a pre-requisite to the application of advanced analytics to the data. Further, big data sets often contain anomalies and errors that are difficult to know a priori. Current approaches to understanding data structure are drawn from the traditional database ontology design. These approaches are effective, but often require too much human involvement to be effective for the volume, velocity and variety of data encountered by big data systems. Dimensional Data Analysis (DDA) is a proposed technique that allows big data analysts to quickly understand the overall structure of a big dataset, determine anomalies. DDA exploits structures that exist in a wide class of data to quickly determine the nature of the data and its statical anomalies. DDA leverages existing schemas that are employed in big data databases today. This paper presents DDA, applies it to a number of data sets, and measures its performance. The overhead of DDA is low and can be applied to existing big data systems without greatly impacting their computing requirements.
Exploratory Analysis for Big Social Data Using Deep Network

Chao Wu,Guolong Wang,Jiangcheng Zhu,Piyawat Lertvittayakumjorn,Simon Hu,Chilie Tan,Hong Mi,Yadan Xu,Jun Xiao

DOI: https://doi.org/10.1109/access.2019.2898238

IF: 3.9

2019-01-01

IEEE Access

Abstract:Exploratory analysis is an important way to gain understanding and find unknown relationships from various data sources, especially in the era of big data. Traditional paradigms of social science data analysis follow the steps of feature selection, modeling, and prediction. In this paper, we propose a new paradigm that does not require feature selection so that data can speak for itself without manually picking out features. Besides, we propose using the deep network as a methodology to explore previously unknown relationships and capture complexity and non-linearity between target variables and a large number of input features for big social data. The new paradigm tends to be a relatively generic approach that can be widely used in different scenarios. In order to validate the feasibility of the paradigm, we use country-level indicators forecasting as a case study. The process includes: 1) data collection and preparation and 2) modeling and experiment. The data collection and preparation part builds a data warehouse and conducts the extract-transform-load process to eliminate data format inconsistency. The modeling and experiment part includes model setup and model structures change to achieve relatively high accuracy on prediction results at both model level and case level. We find some patterns about network capacity modification and the influence of time interval difference on the test results, whereas both of them deserve further research.
Noisy Data Visualization using Functional Data Analysis

Haozhe Chen,Andres Felipe Duque Correa,Guy Wolf,Kevin R. Moon

2024-06-05

Abstract:Data visualization via dimensionality reduction is an important tool in exploratory data analysis. However, when the data are noisy, many existing methods fail to capture the underlying structure of the data. The method called Empirical Intrinsic Geometry (EIG) was previously proposed for performing dimensionality reduction on high dimensional dynamical processes while theoretically eliminating all noise. However, implementing EIG in practice requires the construction of high-dimensional histograms, which suffer from the curse of dimensionality. Here we propose a new data visualization method called Functional Information Geometry (FIG) for dynamical processes that adapts the EIG framework while using approaches from functional data analysis to mitigate the curse of dimensionality. We experimentally demonstrate that the resulting method outperforms a variant of EIG designed for visualization in terms of capturing the true structure, hyperparameter robustness, and computational speed. We then use our method to visualize EEG brain measurements of sleep activity.

Machine Learning,Functional Analysis
Bayesian Data Analysis in Empirical Software Engineering: The Case of Missing Data

Richard Torkar,Robert Feldt,Carlo A. Furia

DOI: https://doi.org/10.1007/978-3-030-32489-6_11

2020-01-01

Abstract:Bayesian data analysis (BDA) is today used by a multitude of research disciplines. These disciplines use BDA as a way to embrace uncertainty by using multilevel models and making use of all available information at hand. In this chapter, we first introduce the reader to BDA and then provide an example from empirical software engineering, where we also deal with a common issue in our field, i.e., missing data. The example we make use of presents the steps done when conducting state-of-the-art statistical analysis. First, we need to understand the problem we want to solve. Second, we conduct causal analysis. Third, we analyze non-identifiability. Fourth, we conduct missing data analysis. Finally, we do a sensitivity analysis of priors. All this before we design our statistical model. Once we have a model, we present several diagnostics one can use to conduct sanity checks. We hope that through these examples, the reader will see the advantages of using BDA. This way, we hope Bayesian statistics will become more prevalent in our field, thus partly avoiding the reproducibility crisis we have seen in other disciplines.
DEA under big data: data enabled analytics and network data envelopment analysis

Joe Zhu

DOI: https://doi.org/10.1007/s10479-020-03668-8

IF: 4.82

2020-06-08

Annals of Operations Research

Abstract:This paper proposes that data envelopment analysis (DEA) should be viewed as a method (or tool) for data-oriented analytics in performance evaluation and benchmarking. While computational algorithms have been developed to deal with large volumes of data (decision making units, inputs, and outputs) under the conventional DEA, valuable information hidden in big data that are represented by network structures should be extracted by DEA. These network structures, e.g., transportation and logistics systems, encompass a broader range of inter-linked metrics that cannot be modelled by conventional DEA. It is proposed that network DEA is related to the value dimension of big data. It is shown that network DEA is different from standard DEA, although it bears the name of DEA and some similarity with conventional DEA. Network DEA is big data enabled analytics (big DEA) when multiple (performance) metrics or attributes are linked through network structures. These network structures are too large or complex to be dealt with by conventional DEA. Unlike conventional DEA that are solved via linear programming, general network DEA corresponds to nonconvex optimization problems. This represents opportunities for developing techniques for solving non-linear network DEA models. Areas such as transportation and logistics system as well as supply chains have a great potential to use network DEA in big data modeling.

operations research & management science
DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python

Jinglin Peng,Weiyuan Wu,Brandon Lockhart,Song Bian,Jing Nathan Yan,Linghao Xu,Zhixuan Chi,Jeffrey M. Rzeszotarski,Jiannan Wang

DOI: https://doi.org/10.1145/3448016.3457330

2021-06-09

Abstract:Exploratory Data Analysis (EDA) is a crucial step in any data science project. However, existing Python libraries fall short in supporting data scientists to complete common EDA tasks for statistical modeling. Their API design is either too low level, which is optimized for plotting rather than EDA, or too high level, which is hard to specify more fine-grained EDA tasks. In response, we propose DataPrep.EDA, a novel task-centric EDA system in Python. DataPrep.EDA allows data scientists to declaratively specify a wide range of EDA tasks in different granularity with a single function call. We identify a number of challenges to implement DataPrep.EDA, and propose effective solutions to improve the scalability, usability, customizability of the system. In particular, we discuss some lessons learned from using Dask to build the data processing pipelines for EDA tasks and describe our approaches to accelerate the pipelines. We conduct extensive experiments to compare DataPrep.EDA with Pandas-profiling, the state-of-the-art EDA system in Python. The experiments show that DataPrep.EDA significantly outperforms Pandas-profiling in terms of both speed and user experience. DataPrep.EDA is open-sourced as an EDA component of DataPrep: https://github.com/sfu-db/dataprep.
Total Empiricism: Learning from Data

Orestis Loukas,Ho Ryun Chung

2023-11-15

Abstract:Statistical analysis is an important tool to distinguish systematic from chance findings. Current statistical analyses rely on distributional assumptions reflecting the structure of some underlying model, which if not met lead to problems in the analysis and interpretation of the results. Instead of trying to fix the model or "correct" the data, we here describe a totally empirical statistical approach that does not rely on ad hoc distributional assumptions in order to overcome many problems in contemporary statistics. Starting from elementary combinatorics, we motivate an information-guided formalism to quantify knowledge extracted from the given data. Subsequently, we derive model-agnostic methods to identify patterns that are solely evidenced by the data based on our prior knowledge. The data-centric character of empiricism allows for its universal applicability, particularly as sample size grows larger. In this comprehensive framework, we re-interpret and extend model distributions, scores and statistical tests used in different schools of statistics.

Statistics Theory,Data Analysis, Statistics and Probability,Methodology
Aggregated Estimating Equation Estimation

Nan Lin,Ruibin Xi

DOI: https://doi.org/10.4310/sii.2011.v4.n1.a8

2011-01-01

Statistics and Its Interface

Abstract:Motivated by the recent active research on online analytical processing (OLAP), we develop a computation and storage efficient algorithm for estimating equation (EE) estimation in massive data sets using a “divide-and-conquer” strategy. In each partition of the data set, we compress the raw data into some low dimensional statistics and then discard the raw data. Then, we obtain an approximation to the EE estimator, the aggregated EE (AEE) estimator, by solving an equation aggregated from the saved low dimensional statistics in all partitions. Such low dimensional statistics are taken as the EE estimates and first-order derivatives of the estimating equations in each partition. We show that, under proper partitioning and some regularity conditions, the AEE estimator is strongly consistent and asymptotically equivalent to the EE estimator. A major application of the AEE technique is to support fast OLAP of EE estimations for data warehousing technologies such as data cubes and data streams. It can also be used to reduce the computation time and conquer the memory constraint problem posed by massive data sets. Simulation studies show that the AEE estimator provides efficient storage and remarkable deduction in computational time, especially in its applications to data cubes and data streams.
Communication Theoretic Data Analytics

Kwang-Cheng Chen,Shao-Lun Huang,Lizhong Zheng,H. Vincent Poor

DOI: https://doi.org/10.1109/jsac.2015.2393471

IF: 16.4

2015-01-01

IEEE Journal on Selected Areas in Communications

Abstract:Widespread use of the Internet and social networks invokes the generation of big data, which is proving to be useful in a number of applications. To deal with explosively growing amounts of data, data analytics has emerged as a critical technology related to computing, signal processing, and information networking. In this paper, a formalism is considered in which data are modeled as a generalized social network and communication theory and information theory are thereby extended to data analytics. First, the creation of an equalizer to optimize information transfer between two data variables is considered, and financial data are used to demonstrate the advantages of this approach. Then, an information coupling approach based on information geometry is applied for dimensionality reduction, with a pattern recognition example to illustrate the effectiveness of this formalism. These initial trials suggest the potential of communication theoretic data analytics for a wide range of applications.
[Apropos of the fatal evolution of 2 cases of hydatid cysts ruptured into the biliary tract].

C. Darnaud,J. Soutoul,R. Voisin,J. Lemozy

1963-05-01

Abstract:
Automating Exploratory Data Analysis via Machine Learning: An Overview

Tova Milo,Amit Somech

DOI: https://doi.org/10.1145/3318464.3383126

2020-06-11

Abstract:Exploratory Data Analysis (EDA) is an important initial step for any knowledge discovery process, in which data scientists interactively explore unfamiliar datasets by issuing a sequence of analysis operations (e.g. filter, aggregation, and visualization). Since EDA is long known as a difficult task, requiring profound analytical skills, experience, and domain knowledge, a plethora of systems have been devised over the last decade in order to facilitate EDA. In particular, advancements in machine learning research have created exciting opportunities, not only for better facilitating EDA, but to fully automate the process. In this tutorial, we review recent lines of work for automating EDA. Starting from recommender systems for suggesting a single exploratory action, going through kNN-based classifiers and active-learning methods for predicting users' interestingness preferences, and finally to fully automating EDA using state-of-the-art methods such as deep reinforcement learning and sequence-to-sequence models. We conclude the tutorial with a discussion on the main challenges and open questions to be dealt with in order to ultimately reduce the manual effort required for EDA.
New Solutions Based on the Generalized Eigenvalue Problem for the Data Collaboration Analysis

Yuta Kawakami,Yuichi Takano,Akira Imakura

2024-04-22

Abstract:In recent years, the accumulation of data across various institutions has garnered attention for the technology of confidential data analysis, which improves analytical accuracy by sharing data between multiple institutions while protecting sensitive information. Among these methods, Data Collaboration Analysis (DCA) is noted for its efficiency in terms of computational cost and communication load, facilitating data sharing and analysis across different institutions while safeguarding confidential information. However, existing optimization problems for determining the necessary collaborative functions have faced challenges, such as the optimal solution for the collaborative representation often being a zero matrix and the difficulty in understanding the process of deriving solutions. This research addresses these issues by formulating the optimization problem through the segmentation of matrices into column vectors and proposing a solution method based on the generalized eigenvalue problem. Additionally, we demonstrate methods for constructing collaborative functions more effectively through weighting and the selection of efficient algorithms suited to specific situations. Experiments using real-world datasets have shown that our proposed formulation and solution for the collaborative function optimization problem achieve superior predictive accuracy compared to existing methods.

Machine Learning,Distributed, Parallel, and Cluster Computing
A Mathematical Modelling Technique for the Analysis of the Dynamics of A Simple Continuous Eda

Bo Yuan,Marcus Gallagher

DOI: https://doi.org/10.1109/cec.2006.1688497

2006-01-01

Abstract:This paper presents some initial attempts to mathematically model the dynamics of a continuous estimation of distribution algorithm (EDA) based on a Gaussian distribution and truncation selection. Case studies are conducted on both unimodal and multimodal problems to highlight the effectiveness of the proposed technique and explore some important properties of the EDA. With some general assumptions, we show that, for ID unimodal problems and with the (mu, lambda) scheme: (1). The behaviour of the EDA is dependent only on the general shape of the test function, rather than its specific form; (2). When initialized far from the global optimum, the EDA has a tendency to converge prematurely; (3). Given a certain selection pressure, there is a unique value for the proposed amplification parameter that could help the EDA achieve desirable performance; for ID multimodal problems: (1). The EDA could get stuck with the (mu, lambda) scheme; (2). The EDA will never get stuck with the (mu, lambda) scheme.
Analysing symbolic data by pseudo-marginal methods

Yu Yang,Matias Quiroz,Boris Beranger,Robert Kohn,Scott A. Sisson

DOI: https://doi.org/10.48550/arXiv.2408.04419

2024-08-08

Abstract:Symbolic data analysis (SDA) aggregates large individual-level datasets into a small number of distributional summaries, such as random rectangles or random histograms. Inference is carried out using these summaries in place of the original dataset, resulting in computational gains at the loss of some information. In likelihood-based SDA, the likelihood function is characterised by an integral with a large exponent, which limits the method's utility as for typical models the integral unavailable in closed form. In addition, the likelihood function is known to produce biased parameter estimates in some circumstances. Our article develops a Bayesian framework for SDA methods in these settings that resolves the issues resulting from integral intractability and biased parameter estimation using pseudo-marginal Markov chain Monte Carlo methods. We develop an exact but computationally expensive method based on path sampling and the block-Poisson estimator, and a much faster, but approximate, method based on Taylor expansion. Through simulation and real-data examples we demonstrate the performance of the developed methods, showing large reductions in computation time compared to the full-data analysis, with only a small loss of information.

Methodology,Computation
Statistical Validity and Consistency of Big Data Analytics: A General Framework

Bikram Karmakar,Indranil Mukhopadhyay

DOI: https://doi.org/10.48550/arXiv.1803.10901

2018-03-29

Abstract:Informatics and technological advancements have triggered generation of huge volume of data with varied complexity in its management and analysis. Big Data analytics is the practice of revealing hidden aspects of such data and making inferences from it. Although storage, retrieval and management of Big Data seem possible through efficient algorithm and system development, concern about statistical consistency remains to be addressed in view of its specific characteristics. Since Big Data does not conform to standard analytics, we need proper modification of the existing statistical theory and tools. Here we propose, with illustrations, a general statistical framework and an algorithmic principle for Big Data analytics that ensure statistical accuracy of the conclusions. The proposed framework has the potential to push forward advancement of Big Data analytics in the right direction. The partition-repetition approach proposed here is broad enough to encompass all practical data analytic problems.

Databases,Distributed, Parallel, and Cluster Computing,Methodology
A Non-Parametric Subspace Analysis Approach with Application to Anomaly Detection Ensembles

Marcelo Bacher,Irad Ben-Gal,Erez Shmueli

DOI: https://doi.org/10.48550/arXiv.2101.04932

IF: 5.414

2021-01-13

Machine Learning

Abstract:Identifying anomalies in multi-dimensional datasets is an important task in many real-world applications. A special case arises when anomalies are occluded in a small set of attributes, typically referred to as a subspace, and not necessarily over the entire data space. In this paper, we propose a new subspace analysis approach named Agglomerative Attribute Grouping (AAG) that aims to address this challenge by searching for subspaces that are comprised of highly correlative attributes. Such correlations among attributes represent a systematic interaction among the attributes that can better reflect the behavior of normal observations and hence can be used to improve the identification of two particularly interesting types of abnormal data samples: anomalies that are occluded in relatively small subsets of the attributes and anomalies that represent a new data class. AAG relies on a novel multi-attribute measure, which is derived from information theory measures of partitions, for evaluating the "information distance" between groups of data attributes. To determine the set of subspaces to use, AAG applies a variation of the well-known agglomerative clustering algorithm with the proposed multi-attribute measure as the underlying distance function. Finally, the set of subspaces is used in an ensemble for anomaly detection. Extensive evaluation demonstrates that, in the vast majority of cases, the proposed AAG method (i) outperforms classical and state-of-the-art subspace analysis methods when used in anomaly detection ensembles, and (ii) generates fewer subspaces with a fewer number of attributes each (on average), thus resulting in a faster training time for the anomaly detection ensemble. Furthermore, in contrast to existing methods, the proposed AAG method does not require any tuning of parameters.
Categorical exploratory data analysis on goodness-of-fit issues

Sabrina Enriquez,Fushing Hsieh

DOI: https://doi.org/10.48550/arXiv.2011.09682

IF: 5.414

2020-11-19

Machine Learning

Abstract:If the aphorism "All models are wrong"- George Box, continues to be true in data analysis, particularly when analyzing real-world data, then we should annotate this wisdom with visible and explainable data-driven patterns. Such annotations can critically shed invaluable light on validity as well as limitations of statistical modeling as a data analysis approach. In an effort to avoid holding our real data to potentially unattainable or even unrealistic theoretical structures, we propose to utilize the data analysis paradigm called Categorical Exploratory Data Analysis (CEDA). We illustrate the merits of this proposal with two real-world data sets from the perspective of goodness-of-fit. In both data sets, the Normal distribution's bell shape seemingly fits rather well by first glance. We apply CEDA to bring out where and how each data fits or deviates from the model shape via several important distributional aspects. We also demonstrate that CEDA affords a version of tree-based p-value, and compare it with p-values based on traditional statistical approaches. Along our data analysis, we invest computational efforts in making graphic display to illuminate the advantages of using CEDA as one primary way of data analysis in Data Science education.

Appropriateness of drugs prescribed by primary care physicians for depressed outpatients.

Visualisation and outlier detection for probability density function ensembles

Towards High-Performance Exploratory Data Analysis (EDA) Via Stable Equilibrium Point

MetaInsight: Automatic Discovery of Structured Knowledge for Exploratory Data Analysis

Big data dimensional analysis

Exploratory Analysis for Big Social Data Using Deep Network

Noisy Data Visualization using Functional Data Analysis

Bayesian Data Analysis in Empirical Software Engineering: The Case of Missing Data

DEA under big data: data enabled analytics and network data envelopment analysis

DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python

Total Empiricism: Learning from Data

Aggregated Estimating Equation Estimation

Communication Theoretic Data Analytics

[Apropos of the fatal evolution of 2 cases of hydatid cysts ruptured into the biliary tract].

Automating Exploratory Data Analysis via Machine Learning: An Overview

New Solutions Based on the Generalized Eigenvalue Problem for the Data Collaboration Analysis

A Mathematical Modelling Technique for the Analysis of the Dynamics of A Simple Continuous Eda

Analysing symbolic data by pseudo-marginal methods

Statistical Validity and Consistency of Big Data Analytics: A General Framework

A Non-Parametric Subspace Analysis Approach with Application to Anomaly Detection Ensembles

Categorical exploratory data analysis on goodness-of-fit issues