Analysing and presenting results

JJ Deeks, JPT Higgins, DG Altman

2005-01-01

Abstract:

What problem does this paper attempt to address?

Experimental design and analysis and their reporting: new guidance for publication in BJP

Michael J Curtis,Richard A Bond,Domenico Spina,Amrita Ahluwalia,Stephen P A Alexander,Mark A Giembycz,Annette Gilchrist,Daniel Hoyer,Paul A Insel,Angelo A Izzo,Andrew J Lawrence,David J MacEwan,Lawrence D F Moon,Sue Wonnacott,Arthur H Weston,John C McGrath

DOI: https://doi.org/10.1111/bph.12856

IF: 7.3

2015-06-26

British Journal of Pharmacology

Abstract:This Editorial is part of a series. To view the other Editorials in this series, visit: http://onlinelibrary.wiley.com/doi/10.1111/bph.12956/abstract; http://onlinelibrary.wiley.com/doi/10.1111/bph.12954/abstract; http://onlinelibrary.wiley.com/doi/10.1111/bph.12955/abstract and http://onlinelibrary.wiley.com/doi/10.1111/bph.13112/abstract

pharmacology & pharmacy
Why we should respect analysis results as data

Joana M Barros,Lukas A Widmer,Mark Baillie,Simon Wandel

DOI: https://doi.org/10.48550/arXiv.2204.09959

2022-04-21

Abstract:The development and approval of new treatments generates large volumes of results, such as summaries of efficacy and safety. However, it is commonly overlooked that analyzing clinical study data also produces data in the form of results. For example, descriptive statistics and model predictions are data. Although integrating and putting findings into context is a cornerstone of scientific work, analysis results are often neglected as a data source. Results end up stored as "data products" such as PDF documents that are not machine readable or amenable to future analysis. We propose a solution to "calculate once, use many times" by combining analysis results standards with a common data model. This analysis results data model re-frames the target of analyses from static representations of the results (e.g., tables and figures) to a data model with applications in various contexts, including knowledge discovery. Further, we provide a working proof of concept detailing how to approach analyses standardization and construct a schema to store and query analysis results.

Computers and Society
Ten Points for High-Quality Statistical Reporting and Data Presentation

Pentti Nieminen

DOI: https://doi.org/10.3390/app10113885

2020-06-03

Applied Sciences

Abstract:Background: Data analysis methods have become an essential part of empirical research papers, especially in health sciences and medical research. It has previously been reported that a noteworthy percentage of articles have flaws in their statistical reporting. Reporting problems have been a long-term issue, and despite continued efforts to improve the situation, improvements have been far from satisfactory. One explanation is an inadequate assessment of statistical reporting during peer review. This communication proposes a short instrument to assess the quality of data analysis reporting in manuscripts and published papers. Method: A checklist-type instrument was developed by selecting and refining items from previous reports about the quality of statistical reporting in medical journals and from published guidelines for reporting and data presentation. Items were pretested and modified during pilot studies. A total of 160 original medical research articles that were published in 4 journals were evaluated to test the instrument. Interrater and intrarater agreements were examined by comparing quality scores assigned to 40 articles published in a psychiatric journal. Results: The data analysis reporting test consists of nine questions that assess the quality of health research from a reader’s perspective. The composed scale has a total score ranging from 0 to 10 and discriminated between journals and study designs. A high score suggested that an article had a good presentation of findings in tables and figures and that the description of analysis methods was helpful to readers. Interrater and intrarater agreements were high. Conclusion: An applicable checklist for quickly testing the statistical reporting quality of manuscripts and published research papers was developed. This instrument aims to improve the quality of empirical research in scientific fields where statistical methods play an important role.
Analyzing survey data in marketing research: A guide for academics and postgraduate students

James M. Crick

DOI: https://doi.org/10.1080/0965254X.2023.2176533

2023-02-09

Journal of Strategic Marketing

Abstract:Numerous marketing scholars have been overly loyal to seminal (and out-dated) analytical processes when using surveys. If this continues, these individuals may receive rejection decisions, as the standards of the so-called publishing game have increased. Accordingly, from an Associate Editor perspective, this commentary provides a contemporary guide (with thirteen stages) on how survey data can be effectively analyzed. This covers a series of basic techniques (like treating missing values, presenting profiling information about samples, and undertaking descriptive statistical analyses), alongside describing some scale purification tools and several complex procedures (e.g., running tests for various forms of reliability and validity, addressing common method variance, and testing for endogeneity bias). Then, some conclusions and methodological recommendations are offered to help authors to advance theory and practice with robust survey data. It is anticipated that these discussion points will facilitate debates among readers of the Journal of Strategic Marketing (and beyond) about the analysis of survey data.
Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel,Barnabas Szaszi,Gustav Nilsonne,Olmo R van den Akker,Casper J Albers,Marcel ALM van Assen,Jojanneke A Bastiaansen,Daniel Benjamin,Udo Boehm,Rotem Botvinik-Nezer,Laura F Bringmann,Niko A Busch,Emmanuel Caruyer,Andrea M Cataldo,Nelson Cowan,Andrew Delios,Noah NN van Dongen,Chris Donkin,Johnny B van Doorn,Anna Dreber,Gilles Dutilh,Gary F Egan,Morton Ann Gernsbacher,Rink Hoekstra,Sabine Hoffmann,Felix Holzmeister,Juergen Huber,Magnus Johannesson,Kai J Jonas,Alexander T Kindel,Michael Kirchler,Yoram K Kunkels,D Stephen Lindsay,Jean-Francois Mangin,Dora Matzke,Marcus R Munafò,Ben R Newell,Brian A Nosek,Russell A Poldrack,Don van Ravenzwaaij,Jörg Rieskamp,Matthew J Salganik,Alexandra Sarafoglou,Tom Schonberg,Martin Schweinsberg,David Shanks,Raphael Silberzahn,Daniel J Simons,Barbara A Spellman,Samuel St-Jean,Jeffrey J Starns,Eric Luis Uhlmann,Jelte Wicherts,Eric-Jan Wagenmakers,Noah N N van Dongen,Ann Dreber

DOI: https://doi.org/10.7554/eLife.72185

IF: 7.7

2021-11-09

eLife

Abstract:Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

biology
Analyses That Inform Policy Decisions.

R. Gutman,D. B. Rubin

DOI: https://doi.org/10.1111/j.1541-0420.2011.01732.x

IF: 1.701

2012-01-01

Biometrics

Abstract:Chi Wang,1,2,∗ Giovanni Parmigiani,3,4 and Francesca Dominici4 1Markey Cancer Center, University of Kentucky, Lexington, Kentucky 40536, U.S.A. 2Department of Biostatistics, College of Public Health, University of Kentucky, Lexington, Kentucky 40536, U.S.A. 3Department of Biostatistics and Computational Biology, Dana-Farber Cancer Institute, Boston, Massachusetts 02115, U.S.A. 4Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts 02115, U.S.A. ∗email: chi.wang@uky.edu
The analysis of repeated measurements: A quantitative research synthesis

Joanne C. Keselman,Lisa M. Lix,H. J. Keselman

DOI: https://doi.org/10.1111/j.2044-8317.1996.tb01089.x

1996-11-01

British Journal of Mathematical and Statistical Psychology

Abstract:Meta‐analytic methods were used to summarize the results of Monte Carlo studies investigating the Type I error and power properties of various univariate and multivariate procedures for testing within‐subjects effects in split‐plot repeated measures designs. Results indicated that all test procedures were generally robust to violations of the multivariate normality assumption, but varied in terms of their Type I error control when the sphericity assumption was not satisfied. For balanced designs, the usual F and ê adjusted F tests (Greenhouse & Geisser, 1959) were generally robust to moderate degrees of covariance heterogeneity, whereas the multivariate procedures were slightly more affected by departures from this assumption. When the design was unbalanced, however, all procedures were sensitive to the presence of heterogeneous covariance matrices, particularly when testing the within‐subjects interaction effect. Power rates varied little as a function of assumption violations. However, this finding may be due to the restricted range of many of the variables included in the meta‐analysis of the power data as well as the strong and overshadowing relationship between the degree of non‐centrality and power rates. For balanced designs, the use of either an ê‐adjusted univariate or a multivariate approach is recommended; for unbalanced designs, researchers should consider adopting one of several robust alternatives that have recently been suggested in the literature.

psychology, experimental, mathematical,mathematics, interdisciplinary applications,statistics & probability
The analysis of repeated measures designs: A review

H. J. Keselman,James Algina,Rhonda K. Kowalchuk

DOI: https://doi.org/10.1348/000711001159357

2001-05-01

British Journal of Mathematical and Statistical Psychology

Abstract:Repeated measures ANOVA can refer to many different types of analysis. Specifically, this vague term can refer to conventional tests of significance, one of three univariate solutions with adjusted degrees of freedom, two different types of multivariate statistic, or approaches that combine univariate and multivariate tests. Accordingly, it is argued that, by only reporting probability values and referring to statistical analyses as repeated measures ANOVA, authors convey neither the type of analysis that was used nor the validity of the reported probability value, since each of these approaches has its own strengths and weaknesses. The various approaches are presented with a discussion of their strengths and weaknesses, and recommendations are made regarding the 'best' choice of analysis. Additional topics discussed include analyses for missing data and tests of linear contrasts.

psychology, experimental, mathematical,mathematics, interdisciplinary applications,statistics & probability
Pay attention to the analysis: common statistical errors in orthodontic Randomised Clinical Trials.

Jadbinder Seehra,Nikolaos Pandis

DOI: https://doi.org/10.1053/j.sodo.2024.01.006

IF: 1.34

2024-01-12

Seminars in Orthodontics

Abstract:The introduction of reporting checklists such as CONSORT aimed to enhance the clarity and transparency of the reporting of Randomised Clinical Trials (RCTs). A benefit of clear reporting is to allow healthcare clinicians to assess both the study quality and trial findings and determine its applicability in the treatment of their patients. Despite this global initiative, the reporting of Randomised Clinical Trials (RCTs) undertaken in both dentistry and orthodontics has been reported to sub-optimal in relation to several domains affecting the validity of the trial. Within the literature the adequacy of the statistical analysis undertaken in a trial has been questioned. The aim of this narrative review is to provide clinicians with an overview of the common statistical errors evident in published orthodontic Clinical Trials and to highlight the potential consequences on the interpretation of the trial findings.

dentistry, oral surgery & medicine
Interpretation of data from multiple trials: a critical review

H. O. CONN

DOI: https://doi.org/10.1046/j.1365-2796.1997.118127000.x

1997-03-01

Journal of Internal Medicine

Abstract:The objective was to review the current status of meta-analysis, the process of combining the results of similar investigations, which was described in 1976. It has evolved rapidly, and thousands have already been published. Meta-analysis may be performed in a narrative form or in the preferred statistical format, which is more rigorous and demanding. The criteria for designing and performing a meta-analysis of high quality are presented and discussed in detail, including selection criteria, quality, biases and heterogeneity, both of component studies and of results. Critical reviews of meta-analyses are presented. Examples are given from my own field--hepatology--and from my own research experience. Although meta-analysis is simple in concept, it is complex in execution. If performed with insufficient insight or attention to detail, it may give ambiguous or erroneous results.

medicine, general & internal
Synthesizing and presenting findings using other methods

Joanne E McKenzie,Sue E Brennan

DOI: https://doi.org/10.1002/9781119536604.ch12

2019-09-20

Abstract:A range of statistical synthesis methods are available, and these may be divided into three categories based on their preferability. Preferable methods are the meta‐analysis methods. This chapter focuses on methods that might be considered when a meta‐analysis of effect estimates is not possible due to incompletely reported data in the primary studies. These methods divide into those that are ‘acceptable’ and ‘unacceptable’. The ‘acceptable’ methods differ in the data they require, the hypotheses they address, limitations around their use, and the conclusions and recommendations that can be drawn. The ‘unacceptable’ methods in common use are described, along with the reasons for why they are problematic. Visual display and presentation of data is especially important for transparent reporting in reviews without meta‐analysis, and should be considered irrespective of whether synthesis is undertaken. Tables and plots structure information to show patterns in the data and convey detailed information more efficiently than text.
How to Use A Subgroup Analysis

Xin Sun,John P. A. Ioannidis,Thomas Agoritsas,Ana C. Alba,Gordon Guyatt

DOI: https://doi.org/10.1001/jama.2013.285063

2014-01-01

Abstract:Clinicians, when trying to apply trial results to patient care, need to individualize patient care and, potentially, manage patients based on results of subgroup analyses. Apparently compelling subgroup effects often prove spurious, and guidance is needed to differentiate credible from less credible subgroup claims. We therefore provide 5 criteria to use when assessing the validity of subgroup analyses: (1) Can chance explain the apparent subgroup effect; (2) Is the effect consistent across studies; (3) Was the subgroup hypothesis one of a small number of hypotheses developed a priori with direction specified; (4) Is there strong preexisting biological support; and (5) Is the evidence supporting the effect based on within- or between-study comparisons. The first 4 criteria are applicable to individual studies or systematic reviews, the last only to systematic reviews of multiple studies. These criteria will help clinicians deciding whether to use subgroup analyses to guide their patient care.
Practical Meta-Analysis

M. Lipsey,David B. Wilson

2000-08-18

Abstract:Introduction Problem Specification and Study Retrieval Selecting, Computing and Coding the Effect Size Statistic Developing a Coding Scheme and Coding Study Reports Data Management Analysis Issues and Strategies Computational Techniques for Meta-Analysis Data Interpreting and Using Meta-Analysis Results

Mathematics,Computer Science
COMPARISON OF 3 MAJOR METAANALYTIC APPROACHES

BT JOHNSON,B MULLEN,E SALAS

DOI: https://doi.org/10.1037/0021-9010.80.1.94

IF: 11.8025

1995-01-01

Journal of Applied Psychology

Abstract:Three major meta-analytic approaches have been developed and applied to integrate empirical research. A comparison of their statistical formulas suggested that they should yield different answers to the meta-analytic questions of central tendency, variability, and prediction by moderators. These frameworks were used to analyze systematically differing databases and showed that, although the techniques of L. V. Hedges and I. Olkin (1985)and of R. Rosenthal and D. Rubin(e.g., 1978, 1988; Rosenthal, 1991)tended to produce reasonable and convergent results, the results of J. E. Hunter, F. L. Schmidt, and G. B. Jackson( 1982; Hunter and Schmidt, 1990) often diverged from the other 2 frameworks. For example, consistent with the law of large numbers, finding more studies with the same result is less likely to occur because of chance alone; the Hedges and Olkin and the Rosenthal and Rubin approaches confirmed this prediction, but the Hunter et al. approach did not. The distinct tendency of the Hunter et al. framework to produce results that violate conventional expectations suggests that it should be used with caution.
Investigation of Two Preliminary Analysis-Altering Elements: Ordinate Scaling and DPPXYR

Corey Peltier,John William McKenna,Wilhelmina van Dijk

DOI: https://doi.org/10.1177/01454455231221289

2024-01-14

Behavior Modification

Abstract:Behavior Modification, Ahead of Print. The purpose of this pre-registered study (Peltier & McKenna) was to conceptually replicate if the truncation of the ordinate and DPPXYR increased analysts' estimation of a functional relation and magnitude of treatment effect. Visual analysts (n = 27) evaluated eight data sets reporting null (n = 2), small (n = 2), moderate (n = 2), and large (n = 2) effects. Each data set was graphed six times with manipulations of the ordinate and DPPXYR, resulting in 48 ABAB graphs. We estimated two separate three-level mixed effect models with variations nested in datasets and nested in participants to evaluate the impact of graph characteristics for (1) confidence in determining a functional relation and (2) the estimated magnitude of the treatment effect. We included ordinate scaling and DPPXYR at level 1 and graph effect size at level 2, including all interactions. Overall, graph manipulation consistently did not impact confidence in a functional relation. Results suggest mixed findings for graph manipulation on the estimated magnitude of the treatment effect. Findings will be couched in current literature and recommendations for graph construction and future research will be discussed.

psychology, clinical
[Routine Practice Data for Health Care Analyses: Part 3 of the Manual]

Christof Veit,Thomas Bierbaum,Simone Wesselmann,Stephanie Stock,Claus-Dieter Heidecke,Christian Apfelbacher,Stefan Benz,Karsten E Dreinhöfer,Michael Hauptmann,Falk Hoffmann,Wolfgang Hoffmann,Thomas Kaiser,Monika Klinkhammer-Schalke,Michael Koller,Tanja Kostuj,Olaf Ortmann,Jochen Schmitt,Holger Schünemann,Max Geraedts

DOI: https://doi.org/10.1055/a-1966-0104

Abstract:Analyses of health and health care (hereafter referred to as "health care analyses") usually aim to make transparent the structures, processes, results and interrelationships of health care and to record the degree to which health care systems and their actors have achieved their goals. Health care-related data are an indispensable source of data for many health care analyses. A prerequisite for the examination of a degree of goal achievement is first of all an agreement on those goals that are to be achieved by the system and its substructures, as well as the identification of the determinants of the achievement of the objectives. Primarily it must be examined how safely, effectively and patient-centred systems, facilities and service providers are operating. It also addresses issues of need, accessibility, utilisation, timeliness, appropriateness, patient safety, coordination, continuity, and health economic efficiency and equity of health care. The results of health care include system services (outputs), on the one hand, and results (outcomes), on the other, whereby the results (patient-reported outcomes) and experiences (patient-reported experiences) reported are of particular importance. Health care analyses answer basic questions of health care research: who does what, when, how, why and with which resources and effects in routine health care. Health care analyses thus provide the necessary findings and key figures to further develop health care in order to improve the quality of health care. The applications range from capacity analyses to following innovations up to the concept of regional and supra-regional monitoring of the quality of care given to the population. Given the progress of digitalisation in Health Care, direct data from the care processes will be increasingly available for health care research. This can support care givers significantly if the findings of the studies are applied precisely and correctly within an adequate methodological frame. This can lead to measurable improved health care quality for patients. Data from the process of health care provision have a high potential. Their use needs the same scientific scrutiny as in all other scientific studies.
Widening the tightrope: new abstract guidelines for BJD authors

A.M. Drucker,J.R. Ingram

DOI: https://doi.org/10.1111/bjd.20498

IF: 11.113

2021-10-01

British Journal of Dermatology

Abstract:Table 1. Items included in the Consolidated Standards of Reporting Trials extension for reporting randomized controlled trials in journal and conference abstractsItemDescriptionTitleIdentification of the study as randomizedAuthorsContact details of supporting authors. This item is specific for conference abstractsTrial designDescription of the trial design (e.g. parallel, cluster, noninferiority)MethodsParticipantsEligibility criteria for participants and the settings where the data were collectedInterventionsInterventions intended for each groupObjectiveSpecific objectives or hypothesisOutcomeClearly defined primary outcome for this reportRandomizationHow participants were allocated to interventionsBlinding (masking)How participants, caregiver and those assessing the outcomes were blinded to group assignmentResultsNumbers randomizedNumber of participants randomized to each groupRecruitmentTrial statusNumbers analysedNumber of participants analysed in each groupOutcomeFor the primary outcome, a result for each group and the estimated effect size and its precisionHarmsImportant adverse events or side-effectsConclusionsGeneral interpretation of the resultsTrial registrationRegistration number and name of trial registerFundingSource of funding Modified from Hopewell et al.1

dermatology
A practical guide to data analysis in general literature reviews

Rebecca Popenoe,Ann Langius-Eklöf,Ewa Stenwall,Anna Jervaeus

DOI: https://doi.org/10.1177/2057158521991949

2021-06-26

Nordic Journal of Nursing Research

Abstract:Academic theses at the bachelor’s and master’s levels in nursing, physiotherapy, occupational therapy, public health and related fields often take the form of general reviews of published research on a relevant clinical issue. While there are many guides to doing literature searches and evaluating article quality, there are to our knowledge no published detailed guides of how to do the actual data analysis in such general literature reviews, particularly that are applicable to students. This article seeks to describe a systematic method of data analysis appropriate for undergraduate research theses, where the data consists of the results from available published research. We present a step-by-step guide with authentic examples and practical tips.

English Else
Publishing Without Perishing: A Guide to the Successful Reporting of Clinical Data

José G B Derraik,Éadaoin M Butler,Kittipan Rerkasem

DOI: https://doi.org/10.1177/1534734619865860

Abstract:Many researchers find writing a scientific manuscript a highly discouraging task. This problem may be partly responsible for the fact that approximately half of completed clinical studies worldwide remain unreported or unpublished. Therefore, we aimed to create a user-friendly guide with helpful recommendations, which are complementary to the many existing reporting guidelines for quantitative clinical data. This article has been prepared with clearly defined subheadings, to facilitate quick identification of any specific sections/topics. We encourage the use of the IMRaD model (ie, Introduction, Methods, Results, and Discussion), providing guidance on the key information required, as well as the dos and don'ts. We also comment briefly on feedback and rejection, proposing the I AM approach (Ignore, Address, and Move on).
Advanced Analytic and Statistical Methods in Health Psychology

Kyra Hamilton,Marta M. Marques,Blair T. Johnson

DOI: https://doi.org/10.1080/17437199.2017.1348905

2017-01-01

Health Psychology Review

Abstract:A pressing issue facing health psychology research, and science in general, is ensuring precision and confidence in findings of empirical studies. With concerns surrounding the replicability of fin...

Analysing and presenting results

Experimental design and analysis and their reporting: new guidance for publication in BJP

Why we should respect analysis results as data

Ten Points for High-Quality Statistical Reporting and Data Presentation

Analyzing survey data in marketing research: A guide for academics and postgraduate students

Consensus-based guidance for conducting and reporting multi-analyst studies

Analyses That Inform Policy Decisions.

The analysis of repeated measurements: A quantitative research synthesis

The analysis of repeated measures designs: A review

Pay attention to the analysis: common statistical errors in orthodontic Randomised Clinical Trials.

Interpretation of data from multiple trials: a critical review

Synthesizing and presenting findings using other methods

How to Use A Subgroup Analysis

Practical Meta-Analysis

COMPARISON OF 3 MAJOR METAANALYTIC APPROACHES

Investigation of Two Preliminary Analysis-Altering Elements: Ordinate Scaling and DPPXYR

[Routine Practice Data for Health Care Analyses: Part 3 of the Manual]

Widening the tightrope: new abstract guidelines for BJD authors

A practical guide to data analysis in general literature reviews

Publishing Without Perishing: A Guide to the Successful Reporting of Clinical Data

Advanced Analytic and Statistical Methods in Health Psychology