Performance comparison of GraalVM, Oracle JDK andOpenJDK for optimization of test suite execution time

Sammanfattning: Testing, when done correctly, is an important part of software development sinceit is a measure of the quality of a software in question. Most of the highly ratedsoftware projects therefore have test suites implemented that include unit tests,integration tests, and other types of tests. However, a challenge regarding the testsuite is that it needs to run each time new code changes are proposed. From thedeveloper’s perspective, it might not always be necessary to run the whole testsuite for small code changes. Previous studies have tried to tackle this probleme.g., by only running a subset of the test suite. This research investigates runningthe whole test suite of Java projects faster, by testing the Java Development Kits(JDKs) GraalVM Enterprise Edition (EE) and Community Edition (CE) againstOracle JDK and OpenJDK for Java 8 and 11. The research used the test suiteexecution time as a metric to compare the JDKs. Another metric that wasconsidered was the test suites number of test cases, used to try and find a breakingpoint for when GraalVM becomes beneficial. The tests were performed on twotest machines, where the first used 20 out of 48 tested projects and the secondused 11 out of 43 projects tested. When looking at the average of five runs,GraalVM EE 11 performed best in 11 out of 18 projects on the first test machine,compared to its closest competitor, and in 7 out of 11 projects on the second testmachine both for JDK 8 and 11. However GraalVM EE 8 did not give anybenefits to the first test machine compared to its competitors, which might indicatethat the hardware plays a vital role in the performance of GraalVM EE 8. Numberof test cases could not be used to determine a breaking point for when GraalVM isbeneficial, but it was observed that GraalVM did not show any benefits forprojects with an execution time of fewer than 39 seconds. It is observed thatGraalVM CE, does not perform well as compared to the other JDKs, and in allcases, its performance is not countable due to less non-satisfied and inefficientbehavior.