An Integrated Tool Chain for Combinatorial Testing of Industrial Control Software

Sammanfattning: Testing is an important activity in software development used to ensure the quality of a product. In industrial practice when developing control software, such as Programmable Logic Controllers (PLCs), software testing is, in some cases, neglected during the development process. There is little support for automated testing on PLC software and the creation of test cases is hence mainly a manual activity that can be both costly and time-consuming. In this thesis, we propose a solution for an integrated tool chain for automating the testing process for IEC 61131-3 PLC software in the integrated development environment (IDE) of CODESYS (Controller Development System), using a combinatorial testing tool (i.e., SEAFOX) for test case generation. Further, we measure the applicability and usefulness of the integrated tool chain in terms of the decision coverage (i.e, Observable Decision Coverage) achieved by the generated test cases. For this purpose, several available tools for testing in CODESYS IDE have been examined as well as solutions on how to integrate the chosen tool (i.e., CfUnit) with SEAFOX. After a solution for the tool chain was implemented and integrated, an experiment was conducted to measure decision coverage for the generated test cases on nine industrial programs using Pairwise, Base choice and Random combinatorial testing techniques. The result of our thesis is a fully integrated tool chain consisting of CODESYS, CfUnit, SEAFOX, and a python script used to combine these, and where the SEAFOX tool was further extended to support additional standard data types. This tool chain can be used to create test cases, generate new input values for these using SEAFOX, automate the additional test case creation and integration with CfUnit in CODESYS IDE, as well as for test execution. The results of the experiment show that test cases generated by this tool chain achieved on average 90% decision coverage or higher regardless of the combinatorial technique used, with the overall average level being 94%. Interestingly enough, Random testing scored higher than both Pairwise and Base choice testing. We present our integrated tool chain as a contribution to the automation of test creation and execution for industrial control software in CODESYS IDE. However, we identified several limitations with testing CODESYS timer blocks and when executing larger test suites that can make it problematic to fully test PLC programs in a real-time simulation environment. These challenges should be investigated further by both researchers and practitioners.