Effectiveness of Exploratory Testing, An empirical scrutiny of the challenges and factors affecting the defect detection efficiency

Sammanfattning: Context: Software testing is an integral part of software development life cycle. To improve the quality of software there are different testing approaches practiced over the years. Traditionally software testing is carried out by following approach focusing on prior test design. While exploratory testing is an approach to test software where the tester does not require to follow a specific test design. But rather, exploratory testing should facilitate the tester in testing the complete system comprehensively. Exploratory testing is seen by some, as a way to conduct simultaneous learning, test design and execution of tests simultaneously. While others point to exploratory testing enabling constant evolution of tests in an easy manner. Objectives: In this study we have investigated the field of exploratory testing in literature and industry to understand its perception and application. Further among the stated claims by practitioners, we selected defect detection efficiency and effectiveness claim for empirical validation through an experiment and survey. Methods: In this study, a systematic literature review, interview, experiment and survey are conducted. In the systematic review a number of article sources are used, including IEEE Xplore, ACM Digital Library, Engineering village, Springer Link, Google Scholar and Books database. The systematic review also includes the gray literature published by the practitioners. The selection of studies was done using two-phase and tollgate approach. A total of 47 references were selected as primary studies. Eight semi-structures interviews were conducted with industry practitioners. Experiment had total 4 iterations and 70 subjects. The subjects were selected from industry and academia. The experimental design used was one factor with two interventions and one response variable. Results: Based on our findings from literature review and interviews, the understanding of exploratory testing has improved over the period but still lacks empirical investigation. The results drawn from experimental and survey data shows that exploratory testing proved effective and efficient in finding more critical bugs in limited time. Conclusions: We conclude that exploratory testing has a lot of potential and much more to offer to testing industry. But more empirical investigation and true facts and figures are required to motivate the testing industry to adapt it. We have reported a number of advantages, disadvantages, challenges and factors in this study. We further investigated the claims stated by the ET practitioners through an experiment and survey. The statistical tests were conducted on the collected data to draw meaningful results. We found statistical significance difference in number of true defects found. Using exploratory testing approach testers found far more defects than test case based testing. Although, there was no statistical significance difference between the two approaches for false defects.