TestDAS: Testing method for dynamically adaptive systems

Abstract

The adaptive behavior of Dynamically Adaptive Systems (DAS), such as Dynamic Software Product Lines (DSPLs), is typically designed using adaptation rules, which are context-triggered actions responsible for features activation and deactivation at runtime. This kind of software should have a correct specification at design time and should be tested to avoid unexpected behavior such as an undesired product configuration at runtime. The use of context and the large number of configurations are challenges related to DAS verification and validation. Therefore, methods and tools supporting these activities are needed to ensure the quality of adaptive systems. The literature addresses different aspects of DAS testing, but few work deals with changes in the software features configuration, and they did not focus on the adaptation rules during the adaptive mechanism testing. Also, there is a lack of formalism to model DAS that allows to reason on the actions triggered by adaptation rules over the DAS features. The focus on the adaptation rules is important because they define the adaptation logic and, thus, they are a potential source of design faults and adaptation failures at runtime. In this thesis, a method called TestDAS is proposed to address these gaps. It involves the model checking approach to identify faults in the adaptation rules design, and the generation of tests for validating the adaptive behavior of DAS. The method is based on a model of the adaptive behavior, called Dynamic Feature Transition System (DFTS), which specifies the DAS configurations and the context changes. Moreover, the TestDAS tool is implemented to support the TestDAS use, and a library called CONTroL is developed to support the test execution. The evaluation of TestDAS is performed using: a mutant analysis to evaluate the effectiveness of the model checking approach in the identification of design faults in DAS; a controlled experiment to compare tests generated by TestDAS with tests specified based on the tester’s experience; and an observational study to assess the feasibility of using the developed tools during the TestDAS activities. The results of the effectiveness evaluation show evidence that TestDAS helps in the identification of faults related to adaptation rules design. The experiment, in turn, provides evidence that TestDAS generates more tests and provides a better coverage of the DAS adaptive behavior than experience-based testing. Lastly, the observational study shows that the TestDAS tool and CONTroL can support the DAS testing and model checking.