Errors in predictor variables

Naomi Altman,Martin Krzywinski

DOI: https://doi.org/10.1038/s41592-023-02119-z

IF: 48

2024-01-04

Nature Methods

Abstract:It is the mark of an educated mind to rest satisfied with the degree of precision that the nature of the subject admits and not to seek exactness where only an approximation is possible. —Aristotle

biochemical research methods

What problem does this paper attempt to address?

Why Can't We Accurately Predict Others' Decisions? Prediction Discrepancy in Risky Decision-Making

Qingzhou Sun,Huanren Zhang,Jing Zhang,Xiaoning Zhang

DOI: https://doi.org/10.3389/fpsyg.2018.02190

IF: 3.8

2018-01-01

Frontiers in Psychology

Abstract:Individuals often fail to accurately predict others' decisions in a risky environment. In this paper, we investigate the characteristics and causes of this prediction discrepancy. Participants completed a risky decision-making task mixed with different domains (gain vs. loss) and probabilities (small vs. large), with some participants making decisions for themselves (the actor) and the others predicting the actors' decisions (the predictor). The results demonstrated a prediction discrepancy: predictions were more risk-averse than the actual decisions over small-probability gains and more risk-seeking over large-probability gains, while these patterns were reversed in the loss domain. Reported and predicted levels of emotional stimulation revealed a pattern that is consistent with the notion of risk-as-feelings and empathy gaps. Mediation analysis provided strong evidence that such prediction discrepancy is driven mainly by the predictor's underestimate of the intensity (not the impact) of the actor's emotional state.
Laypeople's beliefs about the effects of common estimator variables on memory

Anne Cormia,Tyler Shapland,Alaina Rasheed,Kathy Pezdek

DOI: https://doi.org/10.1080/09658211.2020.1868527

Memory

Abstract:This research builds on James Ost's research investigating whether laypeople's beliefs align with those of experts. Recent studies that examined the relationship between high-confidence eyewitness identifications and accuracy proposed that the mechanism underlying this relationship may be based on a knowledge-conditional model. According to this model, the accuracy of a confidence judgment depends on knowledge about factors that affect memory accuracy. However, there has not been a comprehensive assessment of laypeople's knowledge about the effect on memory accuracy of many estimator variables known to influence the accuracy of eyewitnesses, specifically those relevant to research on the relationship between witness confidence and accuracy. This study consists of the development of a 30-item scale to assess laypeople's knowledge of the effect of 10 common estimator variables on memory accuracy from three points of view (POV): Self, Other, and Juror. Across MTurk and undergraduate samples, laypeople's beliefs about the effect of these estimator variables were generally consistent with research findings and did not differ as a function of POV. Additionally, for most estimator variables, participants' beliefs about memory were consistent with results in the confidence-accuracy literature; confidence and identification accuracy appear to be poorly calibrated for estimator variables that people know less about.
A Characterization of Prediction Errors

Christopher Meek

DOI: https://doi.org/10.48550/arXiv.1611.05955

IF: 5.414

2016-11-18

Machine Learning

Abstract:Understanding prediction errors and determining how to fix them is critical to building effective predictive systems. In this paper, we delineate four types of prediction errors and demonstrate that these four types characterize all prediction errors. In addition, we describe potential remedies and tools that can be used to reduce the uncertainty when trying to determine the source of a prediction error and when trying to take action to remove a prediction errors.
Mistakes in Thinking about Cognitive Science and How to Reduce Them

P. N. Johnson-Laird

DOI: https://doi.org/10.1523/eneuro.0380-24.2024

2024-10-05

eNeuro

Abstract:This article allows readers to assess their ability to detect errors in thinking in seven case histories of psychologists' thoughts about cognitive science. It explains the nature of the errors and shows that some of them reflect faulty reasoning. It presents a "model method" to improve reasoning. It is based on the theory of mental models, which gives a general account of how individuals think, both deductively and indicatively, and which postulates that individuals construct mental models of possibilities in the world. The model method enhances both the accuracy and speed of reasoning. The article concludes with some general reflections on the role of knowledge of meanings, the world, and context in thinking.

neurosciences
Motivated Errors

Christine L. Exley,Judd B. Kessler

DOI: https://doi.org/10.1257/aer.20191849

IF: 10.7

2024-04-01

American Economic Review

Abstract:Myriad environments allow for the possibility of confusion. Agents may appeal to such confusion—or the possibility of making an honest mistake—to justify their behavior. In three sets of experiments involving thousands of subjects, we document evidence of such motivated errors. We document this evidence in a simple environment in which the scope for errors is small and in more complex environments in which subjects display correlation neglect and an anchoring bias. (JEL C91, D12, D64, D81, D83, D91)

economics
Error Matters: An Initial Exploration of Elementary School Children's Understanding of Experimental Error

Amy M. Masnick,David Klahr

DOI: https://doi.org/10.1080/15248372.2003.9669683

2003-03-01

Journal of Cognition and Development

Abstract:Error is a pervasive and inescapable aspect of empirical science, and it often plays a causal role in experimental outcomes. But little is known about children's understanding of the causes and consequences of experimental error. In this article, we propose a new framework for characterizing experimental error and we use that framework to guide an empirical assessment of elementary school children's understanding of error, their use of theory and evidence in guiding this understanding, and the role of context in reasoning about error. We found that 2nd- and 4th-grade children could both propose and recognize potential sources of error before they could design unconfounded experiments. They used evidence to guide their reasoning, making predictions and drawing conclusions based on the design of their experiments, and they were sensitive to the context of reasoning: They differentiated the role of error in relative and absolute measurements. Long before children have acquired the formal procedures necessary to control error, they have a surprisingly rich-albeit unsystematic-understanding of its various sources.

psychology, experimental, developmental
Studies in subjective probability l: Prediction of random events

KARL HALVOR TEIGEN

DOI: https://doi.org/10.1111/j.1467-9450.1983.tb00471.x

1983-09-01

Scandinavian Journal of Psychology

Abstract:When students are asked to predict the outcome of a random event, where all alternatives are equally probable (lotteries), they tend to choose central, “representative” values, and avoid extreme ones. In ten informal experiments, it is shown how this pattern of choices is influenced by various procedural and structural changes in the basic task. The results show that guessing behavior can be described as a kind of absolute judgment, subject to grouping, anchoring and context effects. Of the two general prediction heuristics originally proposed by Kahneman & Tversky (1973), “representativeness” applies better than “availability”. In fact, a major strategy of guessing is apparently to eschew numbers with prominent, “non‐random” properties, which at the same time are highly available to the subjects.

psychology, multidisciplinary
The "curse of knowledge" when predicting others' knowledge

Jonathan G Tullis,Brennen Feder

DOI: https://doi.org/10.3758/s13421-022-01382-3

Abstract:To succeed in a social world, we must be able to accurately estimate what others know. For example, teachers must anticipate student knowledge to plan lessons and communicate effectively. Yet one's own knowledge consistently contaminates estimates about others' knowledge. We examine how one's knowledge influences the calibration and resolution of participants' estimates of novices' knowledge. Across four experiments, participants studied trivia questions and estimated the percentage of novice participants who would know the answer across multiple study/estimation rounds. When participants were required to answer the question before estimating what novices would know, studying the facts impaired both the calibration and resolution of the estimates. Studying the facts reduced the validity of one's experiences for predicting novices' knowledge, and estimators utilized their own experiences less when predicting novices' knowledge as they studied. Experimentally reducing reliance on one's own knowledge did not improve the accuracy of estimates. The results suggest that learning impairs the accuracy of judgments of others' knowledge, not because estimators rely too heavily on their own experiences, but because estimators lack diagnostic cues about others' knowledge.
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

Kaitlyn Zhou,Dan Jurafsky,Tatsunori Hashimoto

2023-11-14

Abstract:The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs. Here, we study an aspect of model epistemology: how epistemic markers of certainty, uncertainty, or evidentiality like "I'm sure it's", "I think it's", or "Wikipedia says it's" affect models, and whether they contribute to model failures. We develop a typology of epistemic markers and inject 50 markers into prompts for question answering. We find that LMs are highly sensitive to epistemic markers in prompts, with accuracies varying more than 80%. Surprisingly, we find that expressions of high certainty result in a 7% decrease in accuracy as compared to low certainty expressions; similarly, factive verbs hurt performance, while evidentials benefit performance. Our analysis of a popular pretraining dataset shows that these markers of uncertainty are associated with answers on question-answering websites, while markers of certainty are associated with questions. These associations may suggest that the behavior of LMs is based on mimicking observed language use, rather than truly reflecting epistemic uncertainty.

Computation and Language,Artificial Intelligence
The Representation of Meaningful Precision, and Accuracy

A Mani

2024-10-15

Abstract:The concepts of precision, and accuracy are domain and problem dependent. The simplified numeric hard and soft measures used in the fields of statistical learning, many types of machine learning, and binary or multiclass classification problems are known to be of limited use for understanding the meaningfulness of models or their relevance. Arguably, they are neither of patterns nor proofs. Further, there are no good measures or representations for analogous concepts in the cognition domain. In this research, the key issues are reflected upon, and a compositional knowledge representation approach in a minimalist general rough framework is proposed for the problem contexts. The latter is general enough to cover most application contexts, and may be applicable in the light of improved computational tools available.

Artificial Intelligence,Machine Learning,Logic in Computer Science
On the Unknowable Limits to Prediction

Jiani Yan,Charles Rahal

2024-11-28

Abstract:This short Correspondence critiques the classic dichotomization of prediction error into reducible and irreducible components, noting that certain types of error can be eliminated at differential speeds. We propose an improved analytical framework that better distinguishes epistemic from aleatoric uncertainty, emphasizing that predictability depends on information sets and cautioning against premature claims of unpredictability.

Machine Learning,Artificial Intelligence,Computers and Society,Methodology
The Role of Human Fallibility in Psychological Research: A Survey of Mistakes in Data Management

Marton Kovacs,Rink Hoekstra,Balazs Aczel

DOI: https://doi.org/10.1177/25152459211045930

2021-10-01

Advances in Methods and Practices in Psychological Science

Abstract:Errors are an inevitable consequence of human fallibility, and researchers are no exception. Most researchers can recall major frustrations or serious time delays due to human errors while collecting, analyzing, or reporting data. The present study is an exploration of mistakes made during the data-management process in psychological research. We surveyed 488 researchers regarding the type, frequency, seriousness, and outcome of mistakes that have occurred in their research team during the last 5 years. The majority of respondents suggested that mistakes occurred with very low or low frequency. Most respondents reported that the most frequent mistakes led to insignificant or minor consequences, such as time loss or frustration. The most serious mistakes caused insignificant or minor consequences for about a third of respondents, moderate consequences for almost half of respondents, and major or extreme consequences for about one fifth of respondents. The most frequently reported types of mistakes were ambiguous naming/defining of data, version control error, and wrong data processing/analysis. Most mistakes were reportedly due to poor project preparation or management and/or personal difficulties (physical or cognitive constraints). With these initial exploratory findings, we do not aim to provide a description representative for psychological scientists but, rather, to lay the groundwork for a systematic investigation of human fallibility in research data management and the development of solutions to reduce errors and mitigate their impact.
People underestimate the errors made by algorithms for credit scoring and recidivism prediction but accept even fewer errors

Felix G. Rebitschek,Gerd Gigerenzer,Gert G. Wagner

DOI: https://doi.org/10.1038/s41598-021-99802-y

IF: 4.6

2021-10-11

Scientific Reports

Abstract:Abstract This study provides the first representative analysis of error estimations and willingness to accept errors in a Western country (Germany) with regards to algorithmic decision-making systems (ADM). We examine people’s expectations about the accuracy of algorithms that predict credit default, recidivism of an offender, suitability of a job applicant, and health behavior. Also, we ask whether expectations about algorithm errors vary between these domains and how they differ from expectations about errors made by human experts. In a nationwide representative study (N = 3086) we find that most respondents underestimated the actual errors made by algorithms and are willing to accept even fewer errors than estimated. Error estimates and error acceptance did not differ consistently for predictions made by algorithms or human experts, but people’s living conditions (e.g. unemployment, household income) affected domain-specific acceptance (job suitability, credit defaulting) of misses and false alarms. We conclude that people have unwarranted expectations about the performance of ADM systems and evaluate errors in terms of potential personal consequences. Given the general public’s low willingness to accept errors, we further conclude that acceptance of ADM appears to be conditional to strict accuracy requirements.

multidisciplinary sciences
Error-free milestones in error prone measurements

Dylan S. Small,Paul R. Rosenbaum

DOI: https://doi.org/10.1214/08-AOAS233

2009-10-07

Abstract:A predictor variable or dose that is measured with substantial error may possess an error-free milestone, such that it is known with negligible error whether the value of the variable is to the left or right of the milestone. Such a milestone provides a basis for estimating a linear relationship between the true but unknown value of the error-free predictor and an outcome, because the milestone creates a strong and valid instrumental variable. The inferences are nonparametric and robust, and in the simplest cases, they are exact and distribution free. We also consider multiple milestones for a single predictor and milestones for several predictors whose partial slopes are estimated simultaneously. Examples are drawn from the Wisconsin Longitudinal Study, in which a BA degree acts as a milestone for sixteen years of education, and the binary indicator of military service acts as a milestone for years of service.

Applications
It's not a lie... if you believe it: Narrative analysis of autobiographical memories reveals over-confidence disposition in patients who confabulate

Faith Balshin-Rosenberg,Vanessa Ghosh,Asaf Gilboa

DOI: https://doi.org/10.1016/j.cortex.2024.03.008

IF: 4.644

2024-04-03

Cortex

Abstract:Humans perceive their personal memories as fundamentally true, and although memory is prone to inaccuracies, flagrant memory errors are rare. Some patients with damage to the ventromedial prefrontal cortex (vmPFC) recall and act upon patently erroneous memories (spontaneous confabulations). Clinical observations suggest these memories carry a strong sense of confidence, a function ascribed to vmPFC in studies of memory and decision making. However, most studies of the underlying mechanisms of memory overconfidence do not directly probe personal recollections and resort instead to laboratory-based tasks and contrived rating scales. We analyzed naturalistic word use of patients with focal vmPFC damage (N=18) and matched healthy controls (N=23) while they recalled autobiographical memories using the Linguistic Inquiry and Word Count (LIWC) method. We found that patients with spontaneous confabulation (N=7) tended to over-use words related to the categories of 'certainty' and of 'swearwords' compared to both non-confabulating vmPFC patients (N=11) and control participants. Certainty related expressions among confabulating patients were at normal levels during erroneous memories and were over-expressed during accurate memories, contrary to our predictions. We found no elevation in expressions of affect (positive or negative), temporality or drive as would be predicted by some models of confabulation. Thus, erroneous memories may be associated with subjectively lower certainty, but still exceed patients' report criterion because of a global proclivity for overconfidence. This may be compounded by disinhibition reflected by elevated use of swearwords. These findings demonstrate that analysis of naturalistic expressions of memory content can illuminate global meta-mnemonic contributions to memory accuracy complementing indirect laboratory-based correlates of behavior. Memory accuracy is the result of complex interactions among multiple meta-mnemonic processes such as monitoring, report criteria, and control processes which may be shared across decision-making domains.

behavioral sciences,psychology, experimental,neurosciences
Some Grammatical Errors are Frequent, Others are Important

Leshem Choshen,Ofir Shifman,Omri Abend

DOI: https://doi.org/10.48550/arXiv.2205.05730

2022-05-11

Computation and Language

Abstract:In Grammatical Error Correction, systems are evaluated by the number of errors they correct. However, no one has assessed whether all error types are equally important. We provide and apply a method to quantify the importance of different grammatical error types to humans. We show that some rare errors are considered disturbing while other common ones are not. This affects possible directions to improve both systems and their evaluation.
How Do Expectations Affect Learning About Fundamentals? Some Experimental Evidence

Kieran Marray,Nikhil Krishna,Jarel Tang

DOI: https://doi.org/10.48550/arXiv.2002.07229

2021-07-16

Abstract:We test how individuals with incorrect beliefs about their ability learn about an external parameter (`fundamental') when they cannot separately identify the effects of their ability, actions, and the parameter on their output. Heidhues et al. (2018) argue that learning makes overconfident individuals worse off as their beliefs about the fundamental get less accurate, causing them to take worse actions. In our experiment, subjects take incorrectly-marked tests, and we measure how they learn about the marker's accuracy over time. Overconfident subjects put in less effort, and their beliefs about the marker's accuracy got worse, as they learnt. Beliefs about the proportion of correct answers marked as correct fell by 0.05 over the experiment. We find no effect in underconfident subjects.

General Economics
Teachers' judgment accuracy: A replication check by psychometric meta-analysis

Esther Kaufmann

DOI: https://doi.org/10.1371/journal.pone.0307594

IF: 3.7

2024-07-26

PLoS ONE

Abstract:Teachers' judgment accuracy is a core competency in their daily business. Due to its importance, several meta-analyses have estimated how accurately teachers judge students' academic achievements by measuring teachers' judgment accuracy (i.e., the correlation between teachers' judgments of students' academic abilities and students' scores on achievement tests). In our study, we considered previous meta-analyses and updated these databases and the analytic combination of data using a psychometric meta-analysis to explain variations in results across studies. Our results demonstrate the importance of considering aggregation and publication bias as well as correcting for the most important artifacts (e.g., sampling and measurement error), but also that most studies fail to report the data needed for conducting a meta-analysis according to current best practices. We find that previous reviews have underestimated teachers' judgment accuracy and overestimated the variance in estimates of teachers' judgment accuracy across studies because at least 10% of this variance may be associated with common artifacts. We conclude that ignoring artifacts, as in classical meta-analysis, may lead one to erroneously conclude that moderator variables, instead of artifacts, explain any variation. We describe how online data repositories could improve the scientific process and the potential for using psychometric meta-analysis to synthesize results and assess replicability.

multidisciplinary sciences
Accuracy and Preferences for Legal Error

Murat C Mungan,Marie Obidzinski,Yves Oytana

DOI: https://doi.org/10.1093/aler/ahad006

2024-01-04

American Law and Economics Review

Abstract:Abstract Legal procedures used to determine liability trade-off type-1 errors (e.g., false convictions) against type-2 errors (e.g., false acquittals). After noting that people’s relative preferences for type-1 errors (compared to type-2 errors) appear to be negatively correlated with technological advancements, we study how the accuracy of evidence collection methods may affect the trade-off between these two errors. Counter-intuitively, we find that under some conditions, greater accuracy may result in a higher probability of type-1 error (or type-2 error) maximizing deterrence. Then, assuming both errors are decreasing in accuracy, we characterize the type-1 error that emerges under electoral pressures (when the median voter’s preferences are implemented): convictions occur more often than is socially optimal but less often than is necessary to maximize deterrence. Moreover, as the harm from crime increases, the median voter becomes less tolerant of type-1 errors as the legal system’s accuracy increases. We also show that, because the median voter is less averse toward type-1 errors than the average citizen, an increase in accuracy may reduce welfare.

economics,law

Errors in predictor variables

Why Can't We Accurately Predict Others' Decisions? Prediction Discrepancy in Risky Decision-Making

Laypeople's beliefs about the effects of common estimator variables on memory

A Characterization of Prediction Errors

Mistakes in Thinking about Cognitive Science and How to Reduce Them

Motivated Errors

Error Matters: An Initial Exploration of Elementary School Children's Understanding of Experimental Error

Studies in subjective probability l: Prediction of random events

The "curse of knowledge" when predicting others' knowledge

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

The Representation of Meaningful Precision, and Accuracy

On the Unknowable Limits to Prediction

The Role of Human Fallibility in Psychological Research: A Survey of Mistakes in Data Management

People underestimate the errors made by algorithms for credit scoring and recidivism prediction but accept even fewer errors

Error-free milestones in error prone measurements

It's not a lie... if you believe it: Narrative analysis of autobiographical memories reveals over-confidence disposition in patients who confabulate

Some Grammatical Errors are Frequent, Others are Important

How Do Expectations Affect Learning About Fundamentals? Some Experimental Evidence

Teachers' judgment accuracy: A replication check by psychometric meta-analysis

Accuracy and Preferences for Legal Error