Abstract:Epidemiologic and medical studies often rely on evaluators to obtain measurements of exposures or outcomes for study participants, and valid estimates of associations depends on the quality of data. Even though statistical methods have been proposed to adjust for measurement errors, they often rely on unverifiable assumptions and could lead to biased estimates if those assumptions are violated. Therefore, methods for detecting potential `outlier' evaluators are needed to improve data quality during data collection stage. In this paper, we propose a two-stage algorithm to detect `outlier' evaluators whose evaluation results tend to be higher or lower than their counterparts. In the first stage, evaluators' effects are obtained by fitting a regression model. In the second stage, hypothesis tests are performed to detect `outlier' evaluators, where we consider both the power of each hypothesis test and the false discovery rate (FDR) among all tests. We conduct an extensive simulation study to evaluate the proposed method, and illustrate the method by detecting potential `outlier' audiologists in the data collection stage for the Audiology Assessment Arm of the Conservation of Hearing Study, an epidemiologic study for examining risk factors of hearing loss in the Nurses' Health Study II. Our simulation study shows that our method not only can detect true `outlier' evaluators, but also is less likely to falsely reject true `normal' evaluators. Our two-stage `outlier' detection algorithm is a flexible approach that can effectively detect `outlier' evaluators, and thus data quality can be improved during data collection stage.

Outlier Impact and Accommodation Methods: Multiple Comparisons of Type I Error Rates

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Sensitivity analysis with iterative outlier detection for systematic reviews and meta‐analyses

Asymptotics for Outlier Hypothesis Testing

Comparison of Different Response Time Outlier Exclusion Methods: A Simulation Study

A New Outlier Detection Method Considering Outliers As Model Errors

The Impact of Outliers on Cronbach’s Coefficient Alpha Estimate of Reliability: Ordinal/Rating Scale Item Responses

Extending the Mann–Whitney–Wilcoxon Rank Sum Test to Longitudinal Regression Analysis

Hypothesis testing for detecting outlier evaluators

Outlier Separability Analysis with a Multiple Alternative Hypotheses Test

Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Testing against ordered alternatives in one-way ANOVA model with exponential errors

Analytical method for detecting outlier evaluators

Outlier detection and robust variable selection via the penalized weighted LAD-LASSO method

Comparison of Outlier Detection Methods in NEAT Design

Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics

Outlier Detection in Adaptive Functional-Coefficient Autoregressive Models Based on Extreme Value Theory

The Type Ⅰ Error and Power of the Conventional Statistic Analysis Methods in Minimization

A Nonparametric Global Win Probability Approach to the Analysis and Sizing of Randomized Controlled Trials With Multiple Endpoints of Different Scales and Missing Data: Beyond O'Brien–Wei–Lachin

Universal Sequential Outlier Hypothesis Testing