We must let the research question drive study methods

Brennan C Kahan,Tim P Morris,Suzie Cro

DOI: https://doi.org/10.1136/bmj.q173

2024-01-23

BMJ

Abstract:Estimands help researchers to pinpoint their exact research question, say Brennan C Kahan and colleagues

medicine, general & internal

What problem does this paper attempt to address?

Posing Fundable Questions in Mathematics and Science Education

K. D. King,R. J. Ochsendorf,G. E. A. Solomon,F. C. Sloane

DOI: https://doi.org/10.1007/s10763-020-10088-4

2020-04-20

International Journal of Science and Mathematics Education

Abstract:In this article, the authors describe critical issues related to the pursuit of rigorous and innovative research in the fields of mathematics and science education. The paper is framed to help researchers consider aspects of both their research project and their research proposal. The authors describe features of high-quality and fundable research projects and whether the research is intended to support descriptive, design-oriented, or causal interpretation. They discuss the critical role of grounding proposed research in existing literatures; attending to relevant research from associated fields; and posing research questions that are clear, specific, and feasibly addressable. The goals of the research should cohere with the methodological and analytic design proposed. The authors also discuss the multidisciplinary nature of research that has implications for practice in general and for educational practice more specifically. The authors touch on the characteristics of more and less successful interdisciplinary research teams, especially those that draw from education, cognitive science, methodology, educational psychology, the learning sciences, and relevant STEM disciplinary fields. Examples of STEM education research questions from different genres of research are considered and the authors describe how the questions themselves give rise to specific design choices, methodologies, measures, study samples, and analytical models as well as how they can reflect the disciplinary orientations of the researchers. Implications of these issues for successful educational research grant proposal writing are discussed.

education & educational research
Asking a question

Roberto Kolter

DOI: https://doi.org/10.1128/jb.00050-24

IF: 3.2

2024-06-01

Journal of Bacteriology

Abstract:ABSTRACT While scientific research should be carried out objectively, the choices of questions asked and approaches taken are deeply personal and subjective. I urge individuals to pursue questions they love and to periodically scrutinize the reasons (the philosophies) that drive that love. As a case study, I scrutinize the “whys” behind some of the scientific questions I pursued during my career.

microbiology
Cross-Cultural Research Methods: Review and Prospect

Kwok LEUNG,Fan ZHOU

DOI: https://doi.org/10.3724/sp.j.1041.2010.00041

2010-01-01

Acta Psychologica Sinica

Abstract:The study of cultural phenomena is a major topic for behavioral sciences. Cross-cultural research adopts a comparative perspective in analyzing cultural phenomena, and is confronted with several unique problems in research methods. This paper reviews the major methodological issues in cross-cultural research, covering topics in research design, measurement, and data analysis. From a design perspective, the paper begins with the use of consilience methods to strengthen causal inferences in cross-cultural research. The paper then explores issues of measurement invariance/equivalence in cross-cultural research and the associated analytic procedures. Finally, the paper reviews the application of hierarchical linear modeling in cross-cultural research as well as trends in the development of cross-cultural research methods.
Don't let perfect be the enemy of better: In defense of unparameterized megastudies

Wei Li,Joshua K. Hartshorne

DOI: https://doi.org/10.1017/s0140525x23002121

2024-01-01

Behavioral and Brain Sciences

Abstract:Abstract The target article argues researchers should be more ambitious, designing studies that systematically and comprehensively explore the space of possible experiments in one fell swoop. We argue that while “systematic” is rarely achievable, “comprehensive” is often enough. Critically, the recent popularization of massive online experiments shows that comprehensive studies are achievable for most cognitive and behavioral research questions.

behavioral sciences,neurosciences,psychology, biological
The fundamental importance of method to theory

Rick Dale,Anne S. Warlaumont,Kerri L. Johnson

DOI: https://doi.org/10.1038/s44159-022-00120-5

2022-11-30

Nature Reviews Psychology

Abstract:Many domains of inquiry in psychology are concerned with rich and complex phenomena. At the same time, the field of psychology is grappling with how to improve research practices to address concerns with the scientific enterprise. In this Perspective, we argue that both of these challenges can be addressed by adopting a principle of methodological variety. According to this principle, developing a variety of methodological tools should be regarded as a scientific goal in itself, one that is critical for advancing scientific theory. To illustrate, we show how the study of language and communication requires varied methodologies, and that theory development proceeds, in part, by integrating disparate tools and designs. We argue that the importance of methodological variation and innovation runs deep, travelling alongside theory development to the core of the scientific enterprise. Finally, we highlight ongoing research agendas that might help to specify, quantify and model methodological variety and its implications.
Broad versus narrow research questions in evidence synthesis: a parallel to (and plea for) estimands

Antonio Remiro-Azócar,Anders Gorst-Rasmussen

DOI: https://doi.org/10.1002/jrsm.1741

2024-08-23

Abstract:There has been a transition from broad to more specific research questions in the practice of network meta-analysis (NMA). Such convergence is also taking place in the context of individual registrational trials, following the recent introduction of the estimand framework, which is impacting the design, data collection strategy, analysis and interpretation of clinical trials. The language of estimands has much to offer to NMA, particularly given the "narrow" perspective of treatments and target populations taken in health technology assessment.

Applications
Every problem is embedded in a greater whole

Joachim P Sturmberg,Mathew Mercuri

DOI: https://doi.org/10.1111/jep.14139

2024-09-23

Abstract:Peter Drucker pointed out an important distinction between 'doing things right' and 'doing the right thing', which recognised that all problems are embedded in a context and thus can only be understood within their unique contextual setting. Contemporary research practices in clinical medicine often regards contextual factors as potential confounders that will bias effect estimates and thus must be avoided. However rigorous, research devoid of context ultimately deprives users of understanding of the support factors that make research transferable to policy decisions or managing care of individual patients-it stands in the way of 'doing the right thing' in 'real life' settings. Appreciating that all problems are embedded in a greater context means that one should not ignore their interconnected and interdependent systemic nature, that is, every variable is simultaneously dependent and independent. This is the reason for the cascading effects and feedback loops witnessed in disease progression and policy efforts. We discuss the need for researchers to a-priori consider the context of their research question as well as the structural relationships of the variables under investigation, which in turn provides the basis for choosing the most appropriate research design. We have a moral imperative to first 'do the right thing'-ask questions that address the contextual needs of our patients, and then to 'do it right'-choose the best research method to answer this contextually framed need. Only then will our research efforts have meaningful and lasting impacts on patient care.
Clinical Questions And Real Research

J S Strauss,H Hafez

IF: 17.7

1981-01-01

American Journal of Psychiatry

Abstract:Clinical experiences and the questions they generate are a rich potential source for increased understanding of psychopathology and its treatment. Yet a chasm frequently appears to exist isolating clinical observations from the "real" research viewed as essential to developing and testing their implications. This report suggests that a concept of real research limited to large-sample elaborate studies is too narrow for optimal progress in the field. Research principles for developing a clinical intuition systematically, in the clinical context, are described. The important values as well as limitations of this kind of research are noted.
Confidence in research findings depends on theory

David Gal,Brian Sternthal,Bobby J. Calder

DOI: https://doi.org/10.1017/s0140525x23002261

2024-01-01

Behavioral and Brain Sciences

Abstract:Abstract Almaatouq et al. view the purpose of research is to map variable-to-variable relationships (e.g., the effect of X on Y). They also view theory as this mapping of variable-to-variable relationships rather than an explanation of why the relationships occur. However, it is theory as explanation that allows us to reconcile disparate findings and that should guide application.

behavioral sciences,neurosciences,psychology, biological
Let the data talk: embrace exploratory research

Balazs Aczel

DOI: https://doi.org/10.1038/d41586-024-03826-z

IF: 64.8

2024-11-28

Nature

Abstract:Open-ended research is essential to building solid hypotheses in the social sciences — without it, even the best-planned analyses can fail. Open-ended research is essential to building solid hypotheses in the social sciences — without it, even the best-planned analyses can fail.

multidisciplinary sciences
Distinguishing Between Mediators and Confounders is Important for the Causal Inference in Observational Studies

Zhongheng Zhang

DOI: https://doi.org/10.21037/amj.2019.09.03

2019-01-01

AME Medical Journal

Abstract:Observational studies are common in clinical literature because they are less costly and easy to perform, as compared to the randomized controlled trials (RCT). The limitation of such kind of studies is their inability to draw a causal inference due to potential bias. However, there are many approaches being proposed in recent years for the causal inference in observational studies (1-3). Although these methods cannot replace RCTs for causal inference, analysis in their framework can help to make the causal evidence more reliable. In the following discuss, I would like to discuss some of these points by using a published article.
Conceptual and Methodological Pitfalls in Experimental Studies: An Overview, and the Case of Alzheimer’s Disease

Daniela Puzzo,Fiorenzo Conti

DOI: https://doi.org/10.3389/fnmol.2021.684977

2021-06-15

Frontiers in Molecular Neuroscience

Abstract:The main goal of scientific research is to uncover new knowledge to understand reality. In the field of life sciences, the aim of translational research—to transfer results “from bench to bedside”—has to contend with the problem that the knowledge acquired at the “bench” is often not reproducible at the “bedside,” raising the question whether scientific discoveries truly mirror the real world. As a result, researchers constantly struggle to overcome the dichotomy between methodological problems and expectations, as funding agencies and industries demand expandable and quick results whereas patients, who are uninterested in the epistemological dispute, only ask for an effective cure. Despite the numerous attempts made to address reproducibility and reliability issues, some essential pitfalls of scientific investigations are often overlooked. Here, we discuss some limitations of the conventional scientific method and how researcher cognitive bias and conceptual errors have the potential to steer an experimental study away from the search for the vera causa of a phenomenon. As an example, we focus on Alzheimer’s disease research and on some problems that may have undermined most of the clinical trials conducted to investigate it.

neurosciences
Making Quantitative Research Work: From Positivist Dogma to Actual Social Scientific Inquiry

Michael J. Zyphur,Dean C. Pierides

DOI: https://doi.org/10.1007/s10551-019-04189-6

IF: 6.331

2019-05-29

Journal of Business Ethics

Abstract:Researchers misunderstand their role in creating ethical problems when they allow dogmas to purportedly divorce scientists and scientific practices from the values that they embody. Cortina (J Bus Ethics. <a href="https://doi.org/10.1007/s10551-019-04195-8">https://doi.org/10.1007/s10551-019-04195-8</a>, <a href="/article/10.1007/s10551-019-04189-6#ref-CR200">2019</a>), Edwards (J Bus Ethics. <a href="https://doi.org/10.1007/s10551-019-04197-6">https://doi.org/10.1007/s10551-019-04197-6</a>, <a href="/article/10.1007/s10551-019-04189-6#ref-CR202">2019</a>), and Powell (J Bus Ethics. <a href="https://doi.org/10.1007/s10551-019-04196-7">https://doi.org/10.1007/s10551-019-04196-7</a>, <a href="/article/10.1007/s10551-019-04189-6#ref-CR206">2019</a>) help us clarify and further develop our position by responding to our critique of, and alternatives to, this misleading separation. In this rebuttal, we explore how the desire to achieve the separation of facts and values is unscientific on the very terms endorsed by its advocates—this separation is refuted by empirical observation. We show that positivists like Cortina and Edwards offer no rigorous theoretical or empirical justifications to substantiate their claims, let alone critique ours. Following Powell, we point to how classical pragmatism understands 'purpose' in scientific pursuits while also providing an alternative to the dogmas of positivism and related philosophical positions. In place of dogmatic, unscientific cries about an abstract and therefore always-unobservable 'reality,' we invite all organizational scholars to join us in shifting the discussion about quantitative research towards empirically grounded scientific inquiry. This makes the ethics of actual people and their practices central to quantitative research, including the thoughts, discourses, and behaviors of researchers who are always in particular places doing particular things. We propose that quantitative researchers can thus start to think about their research practices as a kind of work, rather than having the status of a kind of dogma. We conclude with some implications that this has for future research and education, including the relevance of research and research methods.

business,ethics
The primary importance of the research question: Implications for understanding natural versus controlled direct effects and the 'cross-world independence assumption'

Ian Shrier,Etsuji Suzuki

DOI: https://doi.org/10.48550/arXiv.2003.04854

2021-08-09

Abstract:When developing new interventions to minimize the harmful effects of an exposure, investigators usually target the mechanisms that mediate the causal effect of the exposure on the outcome. Predicting the causal effect of these new interventions is generally done through identifying either (1) the controlled direct effect, or (2) the pure (natural) direct effect. In this opinion piece, we use the interventionist approach to discuss how these two approaches answer different questions, and the additional underlying assumptions of each compared to the other. We use a specific example for the development of a new intervention that might reduce the harmful effects of smoking on chronic obstructive pulmonary disease by removing the inhalation of harmful chemicals.

Methodology
Translating questions to estimands in randomized clinical trials with intercurrent events

Mats J. Stensrud,Oliver Dukes

DOI: https://doi.org/10.48550/arXiv.2111.08509

2021-11-16

Abstract:Intercurrent (post-treatment) events occur frequently in randomized trials, and investigators often express interest in treatment effects that suitably take account of these events. A naive conditioning on intercurrent events does not have a straight-forward causal interpretation, and the practical relevance of other commonly used approaches is debated. In this work, we discuss how to formulate and choose an estimand, beyond the marginal intention to treat effect, from the point of view of a decision maker and drug developer. In particular, we argue that careful articulation of a practically useful research question should either reflect decision making at this point in time or future drug development. Indeed, a substantially interesting estimand is a formalization of the (plain English) description of a research question. A common feature of estimands that are practically useful is that they correspond to possibly hypothetical but well-defined interventions in identifiable (sub)populations. To illustrate our points, we consider five examples that were recently used to motivate consideration of principal stratum estimands in clinical trials. In all of these examples, we propose alternative causal estimands, such as conditional effects, sequential regime effects and separable effects, that correspond to explicit research questions of substantial interest. Certain questions require stronger assumptions for identification. However, we highlight that our proposed estimands require less stringent assumptions than estimands commonly targeted in these settings, including principal stratum effects.

Methodology,Applications
Selective attention in hypothesis-driven data analysis

Itai Yanai,Martin Lercher

DOI: https://doi.org/10.1101/2020.07.30.228916

2020-07-31

Abstract:When analyzing the results of an experiment, the mental focus on a specific hypothesis might prevent the exploration of other aspects of the data, effectively blinding one to new ideas. To test this notion, we performed an experiment in which we asked undergraduate students to analyze a fictitious dataset. In addition to being asked what they could conclude from the dataset, half of the students were asked to also test specific hypotheses. In line with our notion, students in the hypothesis-free group were almost 5 times more likely to observe an image of a gorilla when simply plotting the data, a proxy for an initial step towards data analysis. If these findings are representative also of scientific research as a whole, they warrant concern about the current emphasis on hypothesis-driven research, especially in the context of information-rich datasets such as those now routinely created in the biological sciences. Our work provides evidence for a link between the psychological effect of selective attention and hypothesis-driven data analysis, and suggests a hidden cost to having a hypothesis when analyzing a dataset.
When Methods Meet Motives: methodological pluralism in Social Work research

Martine Ganzevles,Daan Andriessen,Tine Van Regenmortel,Jaap van Weeghel

DOI: https://doi.org/10.1007/s11135-021-01161-3

2021-05-13

Abstract:Abstract In Social Work research there is a strong debate on the distinctiveness and methodological quality, and how to address the dilemma of rigour and practice relevance. Given the nature of Social Work the field has developed a characteristic research culture that puts emphasis on giving voice to service users and disseminating research knowledge in practice, especially in a stream of so called practice-based research. However, there is no consensus on how to best contribute to the practice of Social Work through research and at the same time producing rigourous scientific outcomes, resulting in methodological pluralism. Studying the perceptions of Social Work researchers on their role, the aims and values of Social Work research and their research approach, provides insight into the methodological pluralism of Social Work research. Thirty-four professors specialising in practice-based Social Work research participated in a Q methodology study. Q methodology combines qualitative and quantitative methods. It helped reveal and describe divergent views as well as consensus. The analysis led to the identification of three differing viewpoints on Social Work research, which have been given the following denominators: The Substantiator, The Change Agent and The Enlightener. The viewpoints provide researchers in the field of Social Work with a framework in which they can position themselves in the methodological pluralism. Researchers state that the viewpoints are helpful in clarifying perspectives on good research, facilitate the discourse on methodological choices to further develop and strengthen Social Work research as a scientific discipline.
Identifying Assumptions and Research Dynamics

Andrew Ellis,Ran Spiegler

2024-02-29

Abstract:A representative researcher pursuing a question has repeated opportunities for empirical research. To process findings, she must impose an identifying assumption, which ensures that repeated observation would provide a definitive answer to her question. Research designs vary in quality and are implemented only when the assumption is plausible enough according to a KL-divergence-based criterion, and then beliefs are Bayes-updated as if the assumption were perfectly valid. We study the dynamics of this learning process and its induced long-run beliefs. The rate of research cannot uniformly accelerate over time. We characterize environments in which it is stationary. Long-run beliefs can exhibit history-dependence. We apply the model to stylized examples of empirical methodologies: experiments, causal-inference techniques, and (in an extension) ``structural'' identification methods such as ``calibration'' and ``Heckman selection.''

Theoretical Economics
Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect

Michael L. DeKay,Nataliya Rubinchik,Zhaojun Li,Paul De Boeck

DOI: https://doi.org/10.1177/17456916221079611

2022-07-16

Perspectives on Psychological Science

Abstract:Perspectives on Psychological Science, Ahead of Print. A metastudy is a set of many tiny studies (microstudies) created from a much larger collection of possibilities. Metastudies can yield many of the benefits of time-consuming replications and meta-analyses but more efficiently and with greater attention to generalizability and the causal effects of moderators. Statistical precision and power are higher than in studies with the same total sample size but with fewer conditions and more participants per condition. In this article, we describe metastudies and their benefits, demonstrate how to conduct a metastudy using the well-known risky-choice framing effect as an example, and report on the generalizability of that effect. In three metastudies (total N = 2,338), the framing effect generalized well across most of the potential moderators tested, as was expected. Surprisingly, however, the effect was up to twice as large when the certain option was replaced with a slightly risky option; prospect theory predicts the opposite, and fuzzy-trace theory predicts no difference. Metastudies provide a relatively quick and not-so-painful way of examining an effect's generalizability without waiting for a meta-analysis. Both individual labs and multilab networks are encouraged to shift from traditional studies to metastudies.

psychology, multidisciplinary
For Objective Causal Inference, Design Trumps Analysis

Donald B. Rubin

DOI: https://doi.org/10.1214/08-aoas187

2008-01-01

Abstract:For obtaining causal inferences that are objective, and therefore have the best chance of revealing scientific truths, carefully designed and executed randomized experiments are generally considered to be the gold standard. Observational studies, in contrast, are generally fraught with problems that compromise any claim for objectivity of the resulting causal inferences. The thesis here is that observational studies have to be carefully designed to approximate randomized experiments, in particular, without examining any final outcome data. Often a candidate data set will have to be rejected as inadequate because of lack of data on key covariates, or because of lack of overlap in the distributions of key covariates between treatment and control groups, often revealed by careful propensity score analyses. Sometimes the template for the approximating randomized experiment will have to be altered, and the use of principal stratification can be helpful in doing this. These issues are discussed and illustrated using the framework of potential outcomes to define causal effects, which greatly clarifies critical issues.

We must let the research question drive study methods

Posing Fundable Questions in Mathematics and Science Education

Asking a question

Cross-Cultural Research Methods: Review and Prospect

Don't let perfect be the enemy of better: In defense of unparameterized megastudies

The fundamental importance of method to theory

Broad versus narrow research questions in evidence synthesis: a parallel to (and plea for) estimands

Every problem is embedded in a greater whole

Clinical Questions And Real Research

Confidence in research findings depends on theory

Let the data talk: embrace exploratory research

Distinguishing Between Mediators and Confounders is Important for the Causal Inference in Observational Studies

Conceptual and Methodological Pitfalls in Experimental Studies: An Overview, and the Case of Alzheimer’s Disease

Making Quantitative Research Work: From Positivist Dogma to Actual Social Scientific Inquiry

The primary importance of the research question: Implications for understanding natural versus controlled direct effects and the 'cross-world independence assumption'

Translating questions to estimands in randomized clinical trials with intercurrent events

Selective attention in hypothesis-driven data analysis

When Methods Meet Motives: methodological pluralism in Social Work research

Identifying Assumptions and Research Dynamics

Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect

For Objective Causal Inference, Design Trumps Analysis