AutoGraphQL : An automated test generation tool for GraphQL

Sammanfattning: This thesis presents a novel technique to automatically generate test cases that assess the implementation of a GraphQL API. The key conceptual foundation for this technique is to log end-user queries when the system is in production and to turn these actual usages of the API into test cases. These test cases aim at revealing ill-formed responses from the API. Hence, we rely on the API schema to automatically generate test oracles. We implemented the technique in a tool called AutoGraphQL and evaluated our approach with one industrial case study. We generate test cases for Frontapp, the system that manages the public web services of Redeye, a software company in Stockholm, Sweden. We have collected end-user queries on the Frontapp API for 33 days in 2021. Based on these queries and Frontapp’s GraphQL schema, AutoGraphQL generated 24, 049 unit tests, which test 231 different query types and include a total of 1.3M assertions. We ran all the test cases against the Frontapp API and observed that this test suite covers 30% of the code that implements the API and 47% of the schema. A total of 157 test cases fail, revealing the presence of errors in Frontapp. The analysis of the failed test cases confirms that all failing cases but two trigger actual bugs in the system. The two exceptions were due to a network error during the test execution and to a bug that was already fixed, which means that none of them were due to a defect in AutoGraphQL. Out of the 157 error messages, 19 were unique and connected to 8 distinct software defects. We show for the first time in the academic literature that it is feasible to turn actual production GraphQL queries into test cases that cover important parts of the code and the schema, and that are able to reveal actual defects in the system under test. In future work, more experiments are needed to confirm these results with other case studies. One could also investigate how to integrate this test generation technique into a complete DevOps pipeline. This includes test suite minimization and prioritization to use the most efficient test cases in a CI/CD pipeline. Redeye plans to implement AutoGraphQL with all of their applications using GraphQL as an API and add generated test cases to a CI/CD pipeline for that application.