Studying Test Flakiness in Python Projects : Original Findings for Machine Learning

Sammanfattning: Flaky tests can fail or pass without any changes to the code under test. They can destroy developers trust in a test suite and if ignored lead to bugs in released code. This thesis performs an empirical study of the most popular open source Python projects on GitHub. 197 commits with keywords indicating test flakiness are manually inspected and categorized according to the root cause of their flakiness. The results are compared to previous studies of Java projects. Two completely novel flakiness causes are found - precision and training (of a machine learning network). These two causes are almost entirely domain-specific to machine learning projects. Precision flakiness comes from an assertion with a too high or too low threshold. Training flakiness is caused by a faulty training setup of a machine learning network in a test. Most tests in Python projects are found to be flaky due to problems with asynchronous waiting, precision and network.