Abstract:Context: Systematic Literature Review (SLR), as a tool of Evidence-Based Software Engineering (EBSE), has been widely used in Software Engineering (SE). However, for certain topics in SE, especially those that are trendy or industry driven, academic literature is generally scarce and consequently Gray Literature (GL) becomes a major source of evidence. In recent years, the adoption of Gray Literature Review (GLR) or Multivocal Literature Review (MLR) is rising steadily to provide the state-of-the-practice of a specific topic where SLR is not a viable option. Objective: Although some SLR guidelines recommend the use of GL and several MLR guidelines have already been proposed in SE, researchers still have conflicting views on the value of GL and commonly accepted GLR or MLR studies are generally lacking in terms of publication. This experience report aims to shed some light on GLR through a case study that uses SLR and MLR guidelines to conduct a GLR on an emerging topic in SE to specifically answer the questions related to the reasons of using GL, the processes of conducting GL, and the impacts of GL on review results. Method: We retrospect the review process of conducting a GLR on the topic of DevSecOps with reference to Kitchenham's SLR and Garousi's MLR guidelines. We specifically reflect on the processes we had to adapt in order to tackle the challenges we faced. We also compare and contrast our GLR with existing MLRs or GLRs in SE to contextualize our reflections. Results: We distill ten challenges in nine activities of a GLR process. We provide reasons for these challenges and further suggest ways to tackle them during a GLR process. We also discuss the decision process of selecting a suitable review methodology among SLR, MLR and GLR and elaborate the impacts of GL on our review results. Conclusion: Although our experience on GLR is mainly derived from a specific case study on DevSecOps, we conjecture that it is relevant and would be beneficial to other GLR or MLR studies. We also expect our experience would contribute to future GLR or MLR guidelines, in a way similar to how SLR guidelines learned from the SLR experience report a dozen years ago. In addition, other researchers may find our decision making process useful before they conduct their own reviews.

Teaching Literature Reviewing for Software Engineering Research

Paving the Way for Mature Secondary Research: The Seven Types of Literature Review

Literature Reviews for Education and Nursing Graduate Students

Benefits and Drawbacks of a Graduate Course: An Experience Teaching Systematic Literature Review

Teaching Research Design in Software Engineering

The Science of Literature Reviews: Searching, Identifying, Selecting, and Synthesising

Processes, Challenges and Recommendations of Gray Literature Review: an Experience Report.

Software Engineering Education Beyond the Technical: A Systematic Literature Review

Teaching Survey Research in Software Engineering

Literature reviews as independent studies: guidelines for academic practice

Advancing knowledge through literature reviews: ‘what’, ‘why’, and ‘how to contribute’

A Research Landscape of Software Engineering Education

Computational Literature Reviews: Method, Algorithms, and Roadmap

Literature Review in Scientific Research: An Overview

On the pragmatic design of literature studies in software engineering: an experience-based guideline

Secondary Studies in the Academic Context: A Systematic Mapping and Survey

A Review of Meta-ethnographies in Software Engineering

Using Gray Literature to Influence Software Engineering Curricula

Towards Continuous Systematic Literature Review in Software Engineering

Designing the literature review: Historical, narrative, theoretical, integrative, and scoping reviews

Guidelines for including grey literature and conducting multivocal literature reviews in software engineering