Experimental Research in Reducing the Risk of Cognitive Bias in Intelligence Analysis
Martha Whitesmith
DOI: https://doi.org/10.1080/08850607.2019.1690329
2020-01-13
International Journal of Intelligence and CounterIntelligence
Abstract:1 Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction, <i>Report of the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction</i> (Washington, DC: United States Government, 2005).2 Uri Bar-Joseph and Arie W. Kruglanski, "Intelligence Failure and the Need for Cognitive Closure: On the Psychology of the Yom Kippur Surprise," <i>Political Psychology</i>, Vol. 24, No. 1 (2003), pp. 75–99.3 National Commission on Terrorist Attacks Upon the United States, <i>The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks upon the United States</i> (Washington, DC: National Commission on Terrorist Attacks upon the United States, 2004).4 Amos Tversky, and Daniel Kahneman, "Judgement Under Uncertainty: Heuristics and Biases," <i>Science</i>, Vol. 185 (1974), pp. 1124–1131.5 Brant A. Cheikes, Mark J. Brown, Paul E. Lehner, and Leonard Adelman, <i>Confirmation Bias in Complex Analyses</i>, Mitre Technical Report MTR 04B0000017 (Bedford, MA: MITRE, Centre for Integrated Intelligence Systems, 2004), and Martha Whitesmith, "The Efficacy of ACH in Mitigating Serial Position Effects and Confirmation Bias in Intelligence Analysis," <i>Journal of Intelligence and National Security</i> (October 2018). doi:10.1080/02684527.2018.15346406 Alternatively referred to as the Analysis of Competing Hypotheses.7 Solomon E. Asch, "Forming Impressions of Personality," <i>Journal of Abnormal and Social Psychology</i>, Vol. 41 (1946), pp. 258–290. Serial position effects are also referred to as <i>information order bias, serial order effects</i>, and <i>order effects.</i>8 A significant proportion of subsequent research into serial position effects has focused on the impact serial position effects have on short- and long-term memory.9 The most prominent previous meta-analysis into the impact of specific analytical conditions on serial position effects was conducted by Robin M. Hogarth and Hillel J. Einhorn, "Order Effects in Belief Updating: The Belief Adjustment Model," <i>Cognitive Psychology</i>, Vol. 24 (1992), pp. l–55.10 The use of the term meta-analysis in this context does not follow the strict scientific interpretation of the term, which refers to statistical analysis of aggregate data, but refers to a comparison of different populations based on an aggregate of prior experimental research, to provide a theory or model to account for the existing data. The methodology for the meta-analysis used two types of statistical tests in combination with qualitative analysis. Two-sample <i>t</i>-tests of proportions were used to identify if there were statistically significant differences between the proportions of participants exhibiting all types of serial position effects for each sub-factor compared against the overall sample. One-sample <i>t</i>-tests of proportions were used to identify if there were statistically significant majority results for the individual serial position effects of recency and primacy. Qualitative analysis was then employed to identify which of the statistically significant differences were likely to have been caused by moderator variables (where one analytical condition factor effects a second factor). Moderator variable effects were identified by consistent high or low proportions of participants being in a moderator variable condition. This process allowed the identification of the variables that were most likely to have a causal effect on serial position effects, and where genuine causal effects attributable to specific analytical conditions differed between the intelligence analysis and non-intelligence analysis samples. Multinomial logistic regression tests were then used to predict the likelihood of occurrence of individual serial position effects from the factors that were identified as corresponding with a statistically significantly higher or lower risk. The meta-analysis process involved multiple comparisons tests. As such, a Bonferroni correction has been applied to the results. The Bonferroni correction for the serial position effect meta-analysis sets the benchmark for statistical significance at <i>p</i> = < .004. For further details of the meta-analysis methodology, please contact the author.11 The 81 studies are: Vernon L. Allen and Robert S. Feldman, "Tutor Attributions and Attitude as a Function of Tutee Performance," <i>Journal of Applied Social Psychology</i>, Vol. 4, No. 4 (1974), pp. 311–320; Norman H. Anderson, "Test of a Model for Number Averaging Behaviour," <i>Psychonomic Science</i>, Vol. 1 (1964), pp. 191–192; Norman H. Anderson, "Application of a Weighted Average Model to a Psychological Averaging Task," <i>Psychonomic Science</i>, Vol. 8, No. 6 (1967), pp. 227–228; Norman H. Anderson, "Application of a Linear-Serial Model to a Personality-Impression Task Using Serial Presentation," <i>Journal of P <p>-Abstract Truncated-</p>
English Else