Abstract:Objectives: We invited inexperienced clinical researchers to analyze coded health datasets and develop hypotheses. We recorded and analyzed their hypothesis generation process. All the hypotheses generated in the process were rated by the same group of seven experts by using the same metrics. This case study examines the higher quality (i.e., higher ratings) and lower quality of hypotheses and participants who generated them. We characterized the contextual factors associated with the quality of hypotheses. Methods: All participants (i.e., clinical researchers) completed a 2-hour study session to analyze data and generate scientific hypotheses using the think-aloud method. Participants screen activity and audio were recorded and transcribed. These transcriptions were used to measure the time used to generate each hypothesis and to code cognitive events (i.e., cognitive activities used when generating hypotheses, for example, -Seeking for Connection- describes an attempt to draw connections between data points). The hypothesis ratings by the expert panel were used as the quality of the hypotheses during the analysis. We analyzed the factors associated with (1) the five highest and (2) five lowest rated hypotheses and (3) the participants who generated them, including the number of hypotheses per participant, the validity of those hypotheses, the number of cognitive events used for each hypothesis, as well as the participant research experience and basic demographics. Results: Participants who generated the five highest-rated hypotheses used similar lengths of time (difference 3:03), whereas those who generated the five lowest-rated hypotheses used more varying lengths of time (difference 7:13). Participants who generated the five highest-rated hypotheses also utilized slightly fewer cognitive events on average compared to the five lowest-rated hypotheses (4 per hypothesis vs. 4.8 per hypothesis). When we examine the participants (who generated the five highest and five lowest hypotheses) and their total hypotheses generated during the 2-hour study sessions, the participants with the five highest-rated hypotheses again had a shorter range of time per hypothesis on average (0:03:34 vs. 0:07:17). They (with the five highest ratings) used fewer cognitive events per hypothesis (3.498 vs. 4.626). They (with the five highest ratings) also had a higher percentage of valid rate (75.51% vs. 63.63%) and generally had more experience with clinical research. Conclusion: The quality of the hypotheses was shown to be associated with the time taken to generate them, where too long or too short time to generate hypotheses appears to be negatively associated with the hypotheses quality ratings. Also, having more experience seems to positively correlate with higher ratings of hypotheses and higher valid rates. Validity is a quality dimension used by the expert panel during rating. However, we acknowledge that our results are anecdotal. The effect may not be simply linear, and future research is necessary. These results underscore the multi-factor nature of hypothesis generation.

Template for a Hypothesis Description paper

The Hierarchy-of-Hypotheses Approach: A Synthesis Method for Enhancing Theory Development in Ecology and Evolution

A Grammar of Hypotheses for Visualization, Data, and Analysis

A Decentralized Approach to the Formulation of Hypotheses: A Hierarchical Structural Model for a Prion Self-Assembled System

Template for an IMEKO event paper

Formalising Hypothesis Virtues in Knowledge Graphs: A General Theoretical Framework and its Validation in Literature-Based Discovery Experiments

Writing and publishing a scientific paper

Hypothesis testing with e-values

Understanding Statistical Hypothesis Testing: The Logic of Statistical Inference

HyMap: eliciting hypotheses in early-stage software startups using cognitive mapping

Automated Text Analysis of Organic Chemistry Students’ Written Hypotheses

Confidences in Hypotheses

Six personas to adopt when framing theoretical research questions in biology

Interpretable hypothesis tests

What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research

Dispelling five myths about hypothesis testing in biological systematics

HyPhy: hypothesis testing using phylogenies

The quality of data-driven hypotheses generated by inexperienced clinical researchers: A case study

PAPER TEMPLATE

Hypothesis Testing in Econometrics

A workflow for human-centered machine-assisted hypothesis generation: Commentary on Banker et al. (2024).