Comparing the Efficacy of Rapid Review With a Systematic Review in the Software Engineering Field
Carolline Pena,Bruno Cartaxo,Igor Steinmacher,Deepika Badampudi,Deyvson da Silva,Williby Ferreira,Adauto Almeida,Fernando Kamei,Sérgio Soares
DOI: https://doi.org/10.1002/smr.2748
2024-12-05
Journal of Software Evolution and Process
Abstract:This study compares a Rapid Review with a Systematic Review in terms of their methods and findings to understand how a Rapid Review is impacted by optimizing its processes. We conclude that both methods are inherently different and complementary. The time and cost required to conduct a Systematic Review can be prohibitive in expert contexts. Therefore, a Rapid Review may be executed as a preliminary, low‐cost step before deciding to invest in a high‐cost Systematic Review. Context Rapid Reviews are secondary studies aiming to deliver evidence to experts in a more timely manner and with lower costs than traditional literature reviews. Previous studies have shown that experts and researchers are positive toward Rapid Reviews. However, little is known about how Rapid Reviews differ from traditional Systematic Reviews. Objective The goal of this paper is to compare a Rapid Review with a Systematic Review in terms of their methods (e.g., search strategy, study selection, quality assessment, and data extraction) and findings to understand how optimizing the traditional Systematic Review method impacts what we obtain with Rapid Review. Method To achieve this goal, we conducted a Systematic Review with the same research questions answered by a pre‐existing Rapid Review and compared those two studies. Also, we surveyed experts from industry and academia to evaluate the relevance of the findings obtained from both the secondary studies. Results The Rapid Review lasted 6 days, while the Systematic Review took 1 year and 2 months. The main bottlenecks we identified in the Systematic Review are (i) executing the search strategy and (ii) selecting the procedure. Together, they took 10 months. The researchers had to analyze the information from 11,383 papers for the Systematic Review compared with 1973 for the Rapid Review. Still, most (∼ 78%) of the papers included in the Systematic Review were returned by the Rapid Review search, and some papers that could be included were unduly excluded during the Rapid Review's selection procedure. Both secondary studies identified the same number of pieces of evidence (30), but the pieces of evidence are not the same. Conclusion The Rapid Review and Systematic Review results are inherently different and complementary. The time and cost to conduct a Systematic Review can be prohibitive in experts' contexts. Thus, at least in such situations, a Rapid Review may be an adequate choice. Moreover, a Rapid Review may be executed in the experts' context as a previous low‐cost step before deciding to invest in a high‐cost Systematic Review.
computer science, software engineering