Um mapeamento sistemático sobre ferramentas para detecção de flaky tests

Abstract

Flaky tests are tests that behave in a non-deterministic way, that is, they have uncertain results, and may return pass and fail results even when executed in the same code of tests that have not changed. A considerable part of test suite failures are caused by flaky tests. If we consider that the continuous integration model is widely adopted in industrial projects, with thousands of tests, the presence of flaky tests can be even more critical, since during regression tests, a single flaky test can paralyze the entire creation process of builds and releases of updates, causing delays in deliveries. In this study, a systematic mapping was carried out to investigate the tools related to Flaky tests in 37 articles, with the objective of answering 3 research questions. The first research question seeks to show the tools found and available to the community, where 30 tools were returned and which were organized according to their publication or creation dates. The second research question is intended to provide an overview of each of the 30 tools collected, encompassing the objectives, techniques or approaches and the study from which each tool was collected, where it was possible to observe, for example, rerun is the most used technique to support detection. The third and final research question is intended to provide the highest level features of the tools and the causes they address, showing, for example, that a total of 46% of the tools address the test order dependency cause and that 70% of the tools are analyzed in the java language. With the discoveries of this study, it is expected to contribute to players in the Software Testing area, researchers and developers, providing inputs for future studies and a more centralized source of available tools, so that those interested can select one or more tools for future studies or some professional necessity.