Introducing automatic software fault localization in a continuous integration environment

Sammanfattning: In this thesis we investigate the usefulness of neural networks to infer the relationship between co-changes of software modules and test verdicts with the goal of localizing software faults. Data for this purpose was collected from a continuous integration (CI) environment at the telecommunications company Ericsson. The data consisted of test verdicts together with information about which software modules had been changed in a software product since last successful test execution. Using this data, different types of neural network models were trained. Following training, an evaluation of the neural networks’ fault localization and defect prediction capabilities was made. For comparison we also included two statistical approaches for fault localization known as Tarantula and Ochiai in the evaluation. There have been similar studies like this thesis in the domain of software fault localization. However, the previous studies all work on the code level. The contribution of this thesis is to examine if the models used in the previous studies perform well when given a different kind of input, namely co-change information of the software modules making up the software product. One major obstacle with the thesis was that we only had data for the problem of software defect prediction. Because of this we had to evaluate the performance of our software fault localization models on the problem of predicting defects.  The results were that all networks performed poorly when predicting defects. The models achieve an accuracy of around 50% and an AUC score of around 0.5. Interestingly, F-score values can reach as high as 0.632 for some models. However, this is most likely a result of properties of the data set used rather than the models learning a relationship between input and output. Comparing the models to Tarantula and Ochiai suggests that neural networks might be too complicated for our situation. Overall, the results suggest that the neural network approaches we have for fault localization do not work in our setting.