Abstract:Context: Organizations opt for continuous delivery of incremental updates to deal with uncertainty and minimize waste. However, applying continuous engineering (CSE) practices requires a continuous feedback loop with input from customers and end-users. Challenges: It becomes increasingly challenging to apply traditional requirements elicitation and validation techniques with ever-shrinking software delivery cycles. At the same time, frequent deliveries generate an abundance of usage data and telemetry informing engineering teams of end-user behavior. The literature describing how practitioners work with user feedback in CSE, is limited. Objectives: We aim to explore the state of practice related to utilization of user feedback in CSE. Specifically, what practices are used, how, and the shortcomings of these practices. Method: We conduct a qualitative survey and report analysis from 21 interviews in 13 product development companies. We apply thematic and cross-case analysis to interpret the data. Results: Based on our earlier work we suggest a conceptual model of how user feedback is utilized in CSE. We further report the identified challenges with the continuous collection and analysis of user feedback and identify implications for practice. Conclusions: Companies use a combination of qualitative and quantitative methods to infer end-user preferences. At the same time, continuous collection, analysis, interpretation, and use of data in decisions are problematic. The challenges pertain to selecting the right metrics and analysis techniques, resource allocation, and difficulties in accessing vaguely defined user groups. Our advice to practitioners in CSE is to ensure sufficient resources and effort for interpretation of the feedback, which can be facilitated by telemetry dashboards.

Evidence-Based Guidelines for Advancing Continuous Experimentation

A theory of factors affecting continuous experimentation (FACE)

Controlled Experimentation in Continuous Experimentation: Knowledge and Challenges

The Viability of Continuous Experimentation in Early-Stage Software Startups: A Descriptive Multiple-Case Study

Continuous Experimentation and Human Factors An Exploratory Study

Towards Continuous Compounding Effects and Agile Practices in Educational Experimentation

The Automotive Take on Continuous Experimentation: A Multiple Case Study

Pre-Release Experimentation in Indie Game Development: An Interview Survey

User Feedback in Continuous Software Engineering: Revealing the State-of-Practice

Experimentation in Early-Stage Video Game Startups: Practices and Challenges

Flexible Online Repeated Measures Experiment

Meta-experiments: Improving experimentation through experimentation

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

The never-ending story–How companies transition to and sustain continuous software engineering practices

Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education

Best of Three Worlds: Adaptive Experimentation for Digital Marketing in Practice

Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring

Some Statistical and Data Challenges When Building Early-Stage Digital Experimentation and Measurement Capabilities

An architecture for enabling A/B experiments in automotive embedded software

Design Criteria to Architect Continuous Experimentation for Self-Driving Vehicles