Investigando a influência dos test smells na ocorrência de flaky tests

Abstract

Software projects often ignore test code quality. The presence of poor test code structures, called test smells, can identify poor test code quality. The presence of test smells can negatively affect test code maintenance, making the code difficult to understand and costly to maintain, potentially compromising test reliability. Studies have indicated that refactoring test smells can prevent the occurrence of flaky tests. A test is considered flaky when it passes in one execution and fails in another, even without modifying the test code or the production code. The presence of flaky tests in test codes generates insecurity, as they can mask real problems since it is not possible to guarantee that the result provided by a flaky test is true. However, the literature on studies investigating the relationship between test smells and flaky tests is still scarce. Thus, this work carries out a deeper investigation of how the presence of test smells can influence the occurrence of flaky tests. For this purpose, a dataset with 82 projects containing Java test code and at least one flaky test was used. Test smells were detected from this dataset in the projects selected in the dataset. After detection, a manual analysis was performed at the method level to identify whether the method with the presence of flaky tests also had a test smell. The main results of the study were that: (i) the test smells Lazy Test, Sleepy Test, Assertion Roulette and Exception Catching Throwing are more related to the occurrence of flaky tests, and it was identified that Lazy Test is more likely to generate flaky tests of the Order Dependency type; (ii) the test smell Sleepy Test is linked to the flaky test Async Wait and (iii) the most common co-occurrence of test smells in a method with flaky tests is Magic Number Test and Assertion Roulette with a total of 7 occurrences. Another contribution of this work is proposing a catalog with the main test smells most related to flaky tests. This catalog also presents the main co-occurrences of test smells that occur in methods with flaky tests.