Abstract:1 Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction, Report of the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction (Washington, DC: United States Government, 2005).2 Uri Bar-Joseph and Arie W. Kruglanski, "Intelligence Failure and the Need for Cognitive Closure: On the Psychology of the Yom Kippur Surprise," Political Psychology, Vol. 24, No. 1 (2003), pp. 75–99.3 National Commission on Terrorist Attacks Upon the United States, The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks upon the United States (Washington, DC: National Commission on Terrorist Attacks upon the United States, 2004).4 Amos Tversky, and Daniel Kahneman, "Judgement Under Uncertainty: Heuristics and Biases," Science, Vol. 185 (1974), pp. 1124–1131.5 Brant A. Cheikes, Mark J. Brown, Paul E. Lehner, and Leonard Adelman, Confirmation Bias in Complex Analyses, Mitre Technical Report MTR 04B0000017 (Bedford, MA: MITRE, Centre for Integrated Intelligence Systems, 2004), and Martha Whitesmith, "The Efficacy of ACH in Mitigating Serial Position Effects and Confirmation Bias in Intelligence Analysis," Journal of Intelligence and National Security (October 2018). doi:10.1080/02684527.2018.15346406 Alternatively referred to as the Analysis of Competing Hypotheses.7 Solomon E. Asch, "Forming Impressions of Personality," Journal of Abnormal and Social Psychology, Vol. 41 (1946), pp. 258–290. Serial position effects are also referred to as information order bias, serial order effects, and order effects.8 A significant proportion of subsequent research into serial position effects has focused on the impact serial position effects have on short- and long-term memory.9 The most prominent previous meta-analysis into the impact of specific analytical conditions on serial position effects was conducted by Robin M. Hogarth and Hillel J. Einhorn, "Order Effects in Belief Updating: The Belief Adjustment Model," Cognitive Psychology, Vol. 24 (1992), pp. l–55.10 The use of the term meta-analysis in this context does not follow the strict scientific interpretation of the term, which refers to statistical analysis of aggregate data, but refers to a comparison of different populations based on an aggregate of prior experimental research, to provide a theory or model to account for the existing data. The methodology for the meta-analysis used two types of statistical tests in combination with qualitative analysis. Two-sample t-tests of proportions were used to identify if there were statistically significant differences between the proportions of participants exhibiting all types of serial position effects for each sub-factor compared against the overall sample. One-sample t-tests of proportions were used to identify if there were statistically significant majority results for the individual serial position effects of recency and primacy. Qualitative analysis was then employed to identify which of the statistically significant differences were likely to have been caused by moderator variables (where one analytical condition factor effects a second factor). Moderator variable effects were identified by consistent high or low proportions of participants being in a moderator variable condition. This process allowed the identification of the variables that were most likely to have a causal effect on serial position effects, and where genuine causal effects attributable to specific analytical conditions differed between the intelligence analysis and non-intelligence analysis samples. Multinomial logistic regression tests were then used to predict the likelihood of occurrence of individual serial position effects from the factors that were identified as corresponding with a statistically significantly higher or lower risk. The meta-analysis process involved multiple comparisons tests. As such, a Bonferroni correction has been applied to the results. The Bonferroni correction for the serial position effect meta-analysis sets the benchmark for statistical significance at p = < .004. For further details of the meta-analysis methodology, please contact the author.11 The 81 studies are: Vernon L. Allen and Robert S. Feldman, "Tutor Attributions and Attitude as a Function of Tutee Performance," Journal of Applied Social Psychology, Vol. 4, No. 4 (1974), pp. 311–320; Norman H. Anderson, "Test of a Model for Number Averaging Behaviour," Psychonomic Science, Vol. 1 (1964), pp. 191–192; Norman H. Anderson, "Application of a Weighted Average Model to a Psychological Averaging Task," Psychonomic Science, Vol. 8, No. 6 (1967), pp. 227–228; Norman H. Anderson, "Application of a Linear-Serial Model to a Personality-Impression Task Using Serial Presentation," Journal of P -Abstract Truncated-

Experimental Research in Reducing the Risk of Cognitive Bias in Intelligence Analysis

The efficacy of ACH in mitigating serial position effects and confirmation bias in an intelligence analysis scenario

Laypeople's beliefs about the effects of common estimator variables on memory

Imprecise counting of observations in averaging tasks predicts primacy and recency effects

Spearman's law of diminishing returns. A meta-analysis

The stability of cognitive abilities: A meta-analytic review of longitudinal studies.

Intention invention and the affect misattribution procedure: reply to Bar-Anan and Nosek (2012)

The analysis of repeated measurements: A quantitative research synthesis

Psychometric curves reveal changes in bias, lapse rate, and guess rate in an online vigilance task

A common factor underlying individual differences in confirmation bias

Differentiation in social perception: Why later-encountered individuals are described more negatively.

A psychometrics of individual differences in experimental tasks

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Meta-analysis of the age-related positivity effect: Age differences in preferences for positive over negative information.

Two forms of minority-group test bias as psychometric artifacts with an animal model (Rattus norvegicus).

Karl Pearson's meta-analysis revisited

A Meta-Analysis of the Interactive, Additive, and Relative Effects of Cognitive Ability and Motivation on Performance

Influences of past moral behavior on future behavior: A review of sequential moral behavior studies using meta-analytic techniques.

The big three: Accuracy, organization, and retrieval effects of latent semantic attributes.

Kelley's Paradox and strength skewness in research on unconscious mental processes

Stereotype lift and stereotype threat effects on subgroup mean differences for cognitive tests: A meta-analysis of adult samples.