Improving Testing Behavior by Gamifying IntelliJ

Philipp Straubinger,Gordon Fraser

2023-10-17

Abstract:Testing is an important aspect of software development, but unfortunately, it is often neglected. While test quality analyses such as code coverage or mutation analysis inform developers about the quality of their tests, such reports are viewed only sporadically during continuous integration or code review, if they are considered at all, and their impact on the developers' testing behavior therefore tends to be negligible. To actually influence developer behavior, it may rather be necessary to motivate developers directly within their programming environment, while they are coding. We introduce IntelliGame, a gamified plugin for the popular IntelliJ Java Integrated Development Environment, which rewards developers for positive testing behavior using a multi-level achievement system: A total of 27 different achievements, each with incremental levels, provide affirming feedback when developers exhibit commendable testing behavior, and provide an incentive to further continue and improve this behavior. A controlled experiment with 49 participants given a Java programming task reveals substantial differences in the testing behavior triggered by IntelliGame: Incentivized developers write more tests, achieve higher coverage and mutation scores, run their tests more often, and achieve functionality earlier.

Software Engineering

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the problem of insufficient testing behavior in software development. Although testing is an important part of software development, in reality, developers often overlook this part. Specifically:

1. **Limited impact of test quality analysis reports**:
- Although test quality analysis tools such as code coverage and mutation analysis can provide detailed reports on test quality, these reports are usually only occasionally viewed during continuous integration or code review processes, and have little impact on developers' daily testing behaviors.

2. **Developers lack motivation**:
- Even if developers understand the importance of testing, they may still overlook testing due to a lack of internal or external motivation. Testing tasks are often considered monotonous, boring, and challenging, causing many developers to be unwilling to invest enough time and energy in testing.

3. **Existing tools fail to effectively motivate testing behaviors**:
- Current IDEs (Integrated Development Environments) provide support tools for writing and executing tests, but lack effective incentive mechanisms to encourage developers to test frequently.

To solve these problems, the paper proposes a way to directly motivate developers to conduct more and better tests in the development environment through "Gamification". The specific method is to introduce a plugin named IntelliGame in the popular IntelliJ Java IDE, and use a multi - level achievement system to reward developers for good testing behaviors. Experimental results show that this method significantly increases the participants' test frequency, test coverage, and mutation scores, and prompts them to implement functions at an earlier stage.

### Main contributions
- Propose a gamification - based method to motivate and reward developers to test while coding.
- Develop and implement the IntelliGame tool, which integrates 27 multi - level achievements and is directly embedded in the IntelliJ development environment.
- Verify the effect of IntelliGame through controlled experiments, and find that the incentive mechanism significantly improves developers' testing behaviors, including writing more tests, increasing coverage and mutation scores, running tests more frequently, and implementing functions earlier.

Through this method, the paper hopes to fundamentally change developers' testing behaviors, thereby improving the overall quality of software.