Studying the impact of early test termination due to assertion failure on code coverage and spectrum-based fault localization
Loading...
Date
2023-08-15
Authors
Uddin, Md. Ashraf
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
An assertion is commonly used to validate the expected program’s behavior (e.g., if the returned value of a method equals an expected value) in software testing. Although it is a recommended practice to use only one assertion in a single test to avoid code smell (e.g., Assertion Roulette), we observe that it is common to have multiple assertions in a single test. One issue with tests that have multiple assertions is that when the test fails at an early assertion, the test will terminate at that point, and the remaining testing code will not be executed. This, in turn, can potentially reduce the code coverage and the performance of techniques that rely on code coverage information (e.g., spectrum-based fault localization). We refer to such a scenario as early test termination. Understanding the impact of early test termination on test coverage is important for software testing and debugging, particularly for the techniques that rely on coverage information obtained from the testing. In this study, we investigated 207 versions of 6 open-source projects. We found that a non-negligible portion of the failed tests (19.1%) is early terminated due to assertion failure, which leads to the skipping of 15.3% to 60.5% of the test code on average, and a negative impact on testing coverage. To mitigate early test termination, we propose two approaches, i.e., Trycatch (adding a try-catch block surrounding an assertion) and Slicing (slicing a test into a set of independent sub-tests, in which only one assertion and its dependent code are contained). After applying our approaches, the line/branch coverage get improved in 55% of the studied versions. Moreover, Slicing improves the performance of SBFL by 15.1% and 10.66% in terms of Mean First Rank (MFR) for Ochiai and Tarantula, respectively. We also provide actionable suggestions to prevent early test termination, and approaches to mitigate early test termination if it already exists in their project.
Description
Keywords
Coverage, Fault Localization, Assertion Failure, Early Test Termination