Test case prioritization in the context of a video game development project

Sammanfattning: Regression testing is an important part of software development and is integral for finding regression errors caused by code changes or issues with revision control. However, executing all test cases in the test suite is often infeasible. One solution to this problem is to use a prioritization technique. Test prioritization techniques are often utilized to rearrange the execution order of test cases, such that the number of software faults found are maximized as early as possible. A test prioritization heuristic is presented in this report, which uses data regarding a test case’s previous execution history to calculate which test case is most likely to find a failure for the current version of the software. The proposed prioritization strategy uses two different modules. The first module statically prioritizes test cases by calculating a score for each test case, based on previous failure rate and distance of last execution. The second module, which is referred to as the history correlator, compares the test histories of every possible pair of tests and check for similarities in test execution histories. When a test has been executed in the regression testing phase, all test cases with similar test execution histories are given highest priority. The proposed strategies are tested in an experiment, using a software project from EA DICE as the software under test. The experiment presented in this report compares the performance of the history score calculation compared to the performance of random prioritization. The performance of both modules together is also compared to only using the first module, to decide if the history correlator improves the performance of the history score module. The results shows that the history score prioritization displays some potential to prioritize test cases that results in failure, but needs future improvements to provide results in a consistent basis. The addition of the history correlator does not lead to significant improvements in detecting failures.