Investigando a qualidade das refatorações de múltiplas LLMs em código Java

Abstract

Code smells negatively affect code quality, facilitating code degradation and technical debt generation. Therefore, when identified, code smells should be removed through various refactoring techniques. Large Language Models (LLM) are large computational language models with the ability to understand and generate human language. They are trained on vast amounts of data to perform a series of downstream tasks. Studies in the literature have investigated LLMs in code generation and also in refactoring code smells. The present study empirically investigates the quality of code smells refactorings using three LLMs (ChatGPT, Github Copilot, Google Gemini) and collecting developers’ perceptions. Thus, we developed a Java project based on the adaptation of code smells (Data Class (DC), Feature Envy (FE), Long Method (LM) and LPL) from open-source projects and assigned 48 developers to teams, each of which refactored the project using one of the LLMs investigated in this study. We analyzed the quality of the refactorings based on the average values of the quality attributes metrics (Cohesion, Coupling, Complexity, and Inheritance) extracted from the refactored projects and compared the performance of the LLMs. We collected developers’ perceptions through questionnaires and conducted a qualitative analysis using Grounded Theory. The results indicate that, after refactorings by LLMs, all attributes improved in terms of coupling, complexity (Exception: DC), and inheritance (Exception: LM). However, cohesion worsened in all code smells except in FE. The refactoring of LPL showed the greatest improvement (Inheritance: 4.93%), but also the greatest decline (Cohesion: -6.07%). The refactorings of FE were the only ones that improved across all attributes. In comparison with the other LLMs, ChatGPT showed an advantage in improving coupling (for FE and LM), Copilot in cohesion (DC and FE) and inheritance (FE and LPL), and Gemini in complexity (DC and FE). The participants reported parallel refactorings and the insertion of new instances of code smells with emphasis on LPL (8.13 insertions per removal) and FE (0.69 insertions per removal). In the participants’ perception, the LLMs assist: (i) in productivity; (ii) code refactoring; (iii) code analysis; and, (iv) as a support tool. However, there are still deficiencies in the refactoring process with LLMs, such as: (i) the need for specific guidance; (ii) understanding and maintaining the context of what was requested; (iii) preventing the generating low-quality code; and, (iv) avoiding dependence on the LLMs.