User Feedback in Continuous Software Engineering: Revealing the State-of-Practice

Anastasiia Tkalich,Eriks Klotins,Tor Sporsem,Viktoria Stray,Nils Brede Moe,Astri Barbala

2024-10-10

Abstract:Context: Organizations opt for continuous delivery of incremental updates to deal with uncertainty and minimize waste. However, applying continuous engineering (CSE) practices requires a continuous feedback loop with input from customers and end-users. Challenges: It becomes increasingly challenging to apply traditional requirements elicitation and validation techniques with ever-shrinking software delivery cycles. At the same time, frequent deliveries generate an abundance of usage data and telemetry informing engineering teams of end-user behavior. The literature describing how practitioners work with user feedback in CSE, is limited. Objectives: We aim to explore the state of practice related to utilization of user feedback in CSE. Specifically, what practices are used, how, and the shortcomings of these practices. Method: We conduct a qualitative survey and report analysis from 21 interviews in 13 product development companies. We apply thematic and cross-case analysis to interpret the data. Results: Based on our earlier work we suggest a conceptual model of how user feedback is utilized in CSE. We further report the identified challenges with the continuous collection and analysis of user feedback and identify implications for practice. Conclusions: Companies use a combination of qualitative and quantitative methods to infer end-user preferences. At the same time, continuous collection, analysis, interpretation, and use of data in decisions are problematic. The challenges pertain to selecting the right metrics and analysis techniques, resource allocation, and difficulties in accessing vaguely defined user groups. Our advice to practitioners in CSE is to ensure sufficient resources and effort for interpretation of the feedback, which can be facilitated by telemetry dashboards.

Software Engineering

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to explore how to utilize user feedback in the practice of Continuous Software Engineering (CSE) and the challenges it faces. Specifically, the research focuses on:

1. **Data collection practices**: Understand what methods of collecting and applying user feedback are used in CSE.
2. **Deficiencies in practice**: Analyze the specific application scenarios of these methods and the existing problems.

#### Background and Motivation

As the market's demand for rapid delivery of high - quality, cutting - edge solutions continues to increase, more and more companies are adopting CSE practices to deal with uncertainty and minimize waste. However, CSE requires a continuous feedback loop to obtain input from customers and end - users. This brings the following challenges:

- **Limitations of traditional requirements collection and verification techniques**: As the software delivery cycle shortens, traditional methods of collecting and verifying user requirements are becoming increasingly difficult to apply.
- **Data deluge brought by frequent deliveries**: Frequent software releases generate a large amount of user behavior data and telemetry information, which poses new challenges to engineering teams.

#### Research Objectives

Through qualitative surveys and case analyses, this paper aims to answer the following research questions:

- **RQ1: How is user feedback collected and applied in Continuous Software Engineering?**
- Motivation: Explore the actual operation of user feedback collection in the context of CSE. Although there has been a large amount of knowledge accumulation regarding user feedback and data collection techniques in the field of requirements engineering, the extent of application of these practices in CSE has not been fully studied.

- **RQ2: What challenges are faced in utilizing user feedback in CSE?**
- Motivation: Identify the challenges of user feedback collection in the context of CSE. Although previous studies have pointed out potential challenges, the specific impacts of these challenges and their relationships with specific feedback collection techniques or CSE process elements have not been fully explored.

#### Methods

To achieve the research objectives, the author adopted a flexible qualitative research design and conducted 21 interviews involving 13 product development companies. Through thematic analysis and cross - case analysis of these data, the actual application of user feedback in CSE and the challenges it faces were revealed.

#### Results

The research shows that in CSE practices, companies usually combine qualitative and quantitative methods to infer the preferences of end - users. However, continuous data collection, analysis, interpretation, and decision - making still have problems, mainly in the following aspects:

- **Selecting appropriate metrics and techniques**: Determining which indicators and analysis techniques can best reflect user needs is a difficult problem.
- **Resource allocation**: Ensuring that there are sufficient manpower and time to process and interpret user feedback is another challenge.
- **Accessing vaguely - defined user groups**: Establishing contact with user groups that are difficult to clearly define is also an obstacle.

#### Conclusions

The author suggests that CSE practitioners should ensure that sufficient resources and efforts are invested in the interpretation of user feedback, and this process can be simplified by tools such as telemetry dashboards. In addition, combining quantitative and qualitative methods can lead to a more comprehensive understanding of user needs, thereby optimizing product development.

In summary, through empirical research, this paper reveals the current situation of user feedback utilization in CSE and the challenges it faces, and proposes improvement suggestions, providing a basis for further research in related fields.