ARAMIS: uma ferramenta multiagente integrada com LLM Open-Source para apoio à correção de TCCs de estudantes de graduação ARAMIS: a Multi-agent Tool Integrated

Abstract

The correction of Undergraduate Final Projects is a crucial stage in the academic development of undergraduate students. However, this process can be time-consuming and exhausting both for students during their research activities and for advisors during supervision, due to factors such as task overload and insufficiently specific feedback on research content. The automation of scientific writing correction using techniques such as Machine Learning (ML) and Natural Language Processing (NLP) has become increasingly present in students’ daily routines, especially after the emergence of Large Language Models (LLMs). In this study, Academic Review Agents for Methodological Improvements ARAMIS was developed, a tool for the analysis and correction of undergraduate theses in portuguese, composed of three specialized agents: grammatical correction, logical chaining and methodological rigor, integrating an open-source LLM guided by prompt engineering techniques. A comparative analysis approach was adopted to evaluate the feedback generation by proprietary and open-source LLMs, with the objective of selecting a model that operates with a satisfactory trade-off. The proposed solution consisted of integrating the best-performing evaluated open-source model into ARAMIS, developed within the scope of this study, focused on returning the analyzed undergraduate theses feedback in portuguese, being composed by the three agents, which serve as the core pillars of automated revision generation. The tool receives the student’s text, which is processed by the LLM and returns a structured review based on the guidelines defined in the agent’s configurations. In this work, the System Usability Scale (SUS) was employed to assess usability. The experiments were conducted with real users actively engaged in undergraduate thesis writing, and the SUS questionnaire was applied immediately after tool usage. The results demonstrate that ARAMIS achieved a score of 90.5/100, confirming that the application meets undergraduate students’s usability expectations, being useful and achieving precise and targeted feedback.