Investigando as causa de Flaky Tests em automação de Testes UI em projetos open-source

Abstract

Flaky Tests are automated test cases that have a non-deterministic behavior, that is, it is not possible to determine if the test case will pass or fail at each execution. Test cases with this characteristic tend to present random results, success or failure, with each new execution on the same version of the System under Test (SUT) and without changes in its source code. Such behavior tends to be detrimental to the software development process. Each false failure destroys the deterministic relationship between test results and software quality, requiring debugging work on test code and production code to determine whether the failure was caused by system defects or the presence of a textitFlaky Test. This debugging consumes time and manpower and can cause delays in deliveries. Thus, seeking to understand more about the occurrence of Flaky Tests in UI Tests, we propose in this work an empirical analysis of commits related to Flaky Tests of open-souce projects on GitHub , to identify its main causes and applied correction strategies. In all, 123 out of 23 projects were analyzed, generating a total of 8 categories of cause and 7 categories of correction, where Add Waiting, Correction of Logic and Ignored Test appear as the main causes of Flaky Tests in Tests UI and Race Condition, Logic Problems and Test Dependency are pointed out as the main fix categories applied in solving Flaky Tests in UI Tests. It is also possible to identify which correction strategies are most used for each identified cause, where it was identified that: 89% of Flaky Tests caused by a Race Condition are corrected through a Wait Addition; 100% of Flaky Tests caused by Logic Problems and Test Dependency are fixed by applying corrections to the logic applied to the tests. The results pointed out in this work will serve as input for understanding the occurrence of Flaky Tests in UI Tests and will help Test Analysts and testers to develop UI Tests automation projects with higher quality, in addition to providing inputs for future studies.