Abstract:Key messageWe review and propose several methods for identifying possible outliers and evaluate their properties. The methods are applied to a genomic prediction program in hybrid rye.AbstractMany plant breeders use ANOVA-based software for routine analysis of field trials. These programs may offer specific in-built options for residual analysis that are lacking in current REML software. With the advance of molecular technologies, there is a need to switch to REML-based approaches, but without losing the good features of outlier detection methods that have proven useful in the past. Our aims were to compare the variance component estimates between ANOVA and REML approaches, to scrutinize the outlier detection method of the ANOVA-based package PlabStat and to propose and evaluate alternative procedures for outlier detection. We compared the outputs produced using ANOVA and REML approaches of four published datasets of generalized lattice designs. Five outlier detection methods are explained step by step. Their performance was evaluated by measuring the true positive rate and the false positive rate in a dataset with artificial outliers simulated in several scenarios. An implementation of genomic prediction using an empirical rye multi-environment trial was used to assess the outlier detection methods with respect to the predictive abilities of a mixed model for each method. We provide a detailed explanation of how the PlabStat outlier detection methodology can be translated to REML-based software together with the evaluation of alternative methods to identify outliers. The method combining the Bonferroni–Holm test to judge each residual and the residual standardization strategy of PlabStat exhibited good ability to detect outliers in small and large datasets and under a genomic prediction application. We recommend the use of outlier detection methods as a decision support in the routine data analyses of plant breeding experiments.

Newton-type Methods for REML Estimation in Genetic Analysis of Quantitative Traits

Efficient Monte Carlo algorithm for restricted maximum likelihood estimation of genetic parameters

Mixed Linear Model Approaches for Analyzing Genetic Models of Complex Quantitative Traits

Comparison of REML methods for the study of phenome-wide genetic variation

Mapping the genetic architecture of complex traits in experimental populations.

A New Bayesian Automatic Model Selection Approach for Mapping Quantitative Trait Loci under Variance Component Model.

Mixed Model Approaches for Diallel Analysis Based on a Bio-Model.

Robust empirical risk minimization via Newton's method

Restricted maximum likelihood estimation in generalized linear mixed models

Bias in Estimates of Quantitative-Trait–Locus Effect in Genome Scans: Demonstration of the Phenomenon and a Method-of-Moments Procedure for Reducing Bias

New approaches of genetic analysis for quantitative traits and their applications in breeding.

Outlier detection methods for generalized lattices: a case study on the transition from ANOVA to REML

High-dimensional genome-wide association study and misspecified mixed model analysis

Comparison of REML and MCMC for Genetic Variance Component Model of Quantitative Trait in Nuclear Family

Inexact Newton-type Methods for Optimisation with Nonnegativity Constraints

Computing log-likelihood and its derivatives for restricted maximum likelihood methods

Quantitative Trait Loci Identification by Estimating the Genetic Model Based on the Extremal Samples.

Robust model-based estimation for binary outcomes in genomics studies

Marker-assisted prediction of non-additive genetic values

Radial Basis Function Regression Methods for Predicting Quantitative Traits Using SNP Markers

Bayesian marker-based principal component ridge regression - a flexible multipurpose framework for quantitative genetics in wild study systems