Behavioural comparison of test and field executions of video games

Sammanfattning: Software developers write tests to ensure that their application behaves as intended. However, the behaviour of an application in production may deviate from the behaviours that are tested. This difference between test and field executions is rarely assessed in practice. Yet, it can be useful in identifying opportunities for the improvement of the test suite. In this thesis, we propose a technique for monitoring the test and field executions of video games. We derive three models to represent the observed behaviours. The first model is method coverage, which enumerates the methods invoked by the test suite, as well as those triggered as a result of player interactions in production. The second model, temporal call chains, identifies the sequences of method calls, in the order of invocation, that arise from test and field executions. The third model, stack call chains, gives additional context into the distinct ways in which each method is triggered by the test suite and in the field. We implement our technique for observing test and field executions of video games in a tool called Ailinea. Ailinea is a video game mod that injects custom instrumentation into methods within C# games, in order to monitor their invocation. We use Ailinea with two popular, real-world video games: Demeo which is developed by Resolution Games, and Osu!, which is open-source. We monitor the execution of Demeo and Osu! with Ailinea in both test and field environments. We successfully derive the three models from the field and test behaviours of the two games, captured by Ailinea. Moreover, all three models identify untested software behaviours, that represent opportunities for improvement of the test suite of each game. In total, we find 139 untested methods, as well as 5, 108 temporal chains and 371 stack chains that occur in the field but are not exercised by the test suites of the two games. Furthermore, in order to automatically propose new test scenarios for Demeo, we extend Ailinea to capture game events in the field, in addition to the stack call chains. Our interview with three Demeo developers indicates that the information contained in these test scenarios can facilitate test suite improvement. The use of mods for runtime observability of video games can have applications such as automated test generation and detection of invalid video game states, paving the way for further work in this direction.