Classificação automática de funcionalidades em postagens relacionadas ao uso

Abstract

The systems evaluation and how it supports experts in the development and improvement of systems design is very important. One type of evaluation that has been gaining prominence in recent publications is textual evaluation, which acts strongly in the area of linguistics. In this type of evaluation, users write, both on social networks and in app stores, about the systems they are using; this is called PRU - Post Related to Use, and its main feature is the user’s spontaneity. Many researches have been made in this area, with topics ranging from the analysis of feelings and temporal analysis, to the automatic classification of facets of usability and automatic identification of PRUs. In this works we present a strategy for the automatic identification of functionalities in PRUs, using NLP techniques - Natural Language Processing - and the adoption of the Model for Evaluation of the interaction in Social Systems from the Textual Language of the User - MALTU. For this, the first part we developed experiments using a set of 200 PRUs from two different databases - 100 from SIGAA and 100 from the MyFitnessPal application, in which these PRUs were pre-processed with text normalization techniques, stopwords remotion and stemming, and then tests were carried out with morphosyntactic analyzers to observe how they perform the linguistic labeling of the functionalities of the PRUs, allowing to generate 3 hypotheses. In the second part, we created an algorithm from the patterns that we identified in the first part was validated, in addition to the validation of the hypotheses. For this, 6 databases were used, containing PRUs, with and without functionality, from 6 different systems, namely: Google Assistant, Waze, Spotify, Google Maps, SIGAA and MyFitnessPal, these last two were used as a comparison and to raise some observations, since they were used in the initial experiments. We also adopted metrics of accuracy, precision, recall and f-measure to evaluate the performance of the algorithm. However, the algorithm correctly classified 48.2% of the base of Google Assistant, 52.6% of the base of Waze, 40% of the base of Spotify and 56.5% of the Google Maps base, with the majority of these PRUs lacking any functionality. Therefore, the evaluation metrics and the hypotheses raised were refuted. Even so, this work was able to contribute strongly to the areas studies of textual evaluation and computational linguistic assessment, presenting a strategy to identify the funcionalities in PRUs, elaborating rules that allow to develop an algorithm, that can be adapted to add new rules, in which this algorithm receives posts and returns results from a classification, in metrics such as accuracy and precision, and finally, investigating how users write about the use of systems in critical type PRUs.