Playstyle Generation with Multimodal Generative Adversarial Imitation Learning : Style-reward from Human Demonstration for Playtesting Agents

Sammanfattning: Playtesting plays a crucial role in video game production. The presence of gameplay issues and faulty design choices can be of great detriment to the overall player experience. Machine learning has the potential to be applied to automated playtesting solutions, removing mundane and repetitive testing, and allowing game designers and playtesters to focus their efforts on rewarding tasks. It is important in playtesting to consider the different playstyles players might use to adapt game design choices accordingly. With Reinforcement learning, it is possible to create high quality agents able to play and traverse complex game environments with fairly simple task-rewards. However, an automated playtesting solution must also be able to incorporate unique behaviour which mimic human playstyles. It can often be difficult to handcraft a quantitative style-reward to drive agent learning, especially for those with limited reinforcement learning experience, like game developers. MultiGAIL, Multimodal Generative Adversarial Imitation Learning, is a proposed learning algorithm able to generate autonomous agents imbued with human playstyles from recorded playstyle demonstrations. The proposed method requires no handcrafted style-reward, and can generate novel intermediate playstyles from demonstrated ones. MultiGAIL is evaluated in game environments resembling complex 3D games with both discrete and continuous action spaces. The playstyle the agent exhibits is easily controllable at inference with an auxiliary input parameter. Evaluation shows the agent is able to successfully replicate the underlying playstyles in human demonstrations, and that novel playstyles generate explainable action distributions indicative of the level of blending the auxiliary input declares. The results indicate that MultiGAIL could be a suitable solution to incorporate style behaviours in playtesting autonomous agents, and can be easily be used by those with limited domain knowledge of reinforcement learning.