Pre-Release Experimentation in Indie Game Development: An Interview Survey

Johan Linåker,Elizabeth Bjarnason,Fabian Fagerholm

2024-11-26

Abstract:[Background] The game industry faces fierce competition and games are developed on short deadlines and tight budgets. Continuously testing and experimenting with new ideas and features is essential in validating and guiding development toward market viability and success. Such continuous experimentation (CE) requires user data, which is often limited in early development stages. This challenge is further exacerbated for independent (indie) game companies with limited resources. [Aim] We wanted to gain insights into CE practices in pre-release indie game development. [Method] We performed an exploratory interview survey with 10 indie game developers from different companies and synthesised findings through an iterative coding process. [Results] We present a CE framework for game development that highlights key parts to consider when planning and implementing an experiment and note that pre-release experimentation is centred on qualitative data. Time and resource constraints impose limits on the type and extent of experimentation and playtesting that indie companies can perform, e.g. due to limited access to participants, biases and representativeness of the target audience. [Conclusions] Our results outline challenges and practices for conducting experiments with limited user data in early stages of indie game development, and may be of value also for larger game companies, and for software intensive organisations in other industries.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the pre - release stage of indie game development, how to effectively conduct Continuous Experimentation (CE) to verify and guide the development direction and ensure that the game is feasible and has the potential for success in the market. Specifically, the paper focuses on how to use user data for effective experiments and tests under limited resources.

### Background
The game industry is highly competitive, with short development cycles and tight budgets. To ensure the technical and commercial feasibility of games, developers need to constantly test and experiment with new ideas and features. However, in the early development stage, user data is usually very limited, which is especially difficult for resource - limited indie game companies.

### Objective
The objective of the paper is to understand how indie game companies conduct continuous experiments in the pre - release stage and summarize an experimental framework applicable to such companies. In this way, it can help indie game companies better utilize limited user data in the early development stage to conduct effective experiments and tests, thereby increasing the success rate of games.

### Method
The researchers conducted semi - structured interview surveys with 10 developers from different indie game companies and comprehensively analyzed the results through an iterative coding process. The researchers hope to extract key practices and challenges regarding continuous experiments from these interviews.

### Results
The research results summarize a continuous experiment framework applicable to game development, emphasizing five key parts that need to be considered when planning and implementing experiments:
1. **Objective Definition**: Clearly define the purpose of the experiment and the aspects of the game to be evaluated.
2. **Design Strategy**: Select appropriate experimental methods (such as A/B testing, sequential testing, exploratory testing, etc.).
3. **Experimental Object**: Determine the maturity of the artworks or prototypes used for the experiment.
4. **Sampling Strategy**: Select the user group participating in the experiment, considering its representativeness and the risk of bias.
5. **Execution Strategy**: Formulate the schedule and frequency of the experiment.

### Conclusion
The research results point out the challenges faced when conducting experiments in the early stage, such as time and resource limitations and the limitations of user feedback. In addition, the study also proposes a preliminary continuous experiment framework, providing a valuable reference for indie game companies. This framework is not only helpful for indie game companies but may also be inspiring for large - scale game companies and other software - intensive organizations.

Through this framework, developers can conduct experiments more effectively under limited resources, thereby better guiding game development and increasing the market competitiveness of games.