ReAvaLIA: um avaliador baseado em Modelo de Linguagem de Grande Escala para redações do Enem

Abstract

ChatGPT’s public release in November 2022 brought to light – and concern – the use of Artificial Intelligence (AI) solutions for human beings and the human substitution in intellectual jobs, subject to errors or not. In most cases, although this substitution could create unemployment and social inequality, AI solutions would develop advantages for many educational processes, including essay evaluations. This task, in addition of being very time-consuming and high priced for teachers and educational organizations, is victimized by human error and its evaluator’s lack of point-of-view about text quality, problems that a highly trained artificial intelligence would easily accomplish. Concerning these possibilities and challenges, we constructed a corpus of essays to test the performance of the ChatGPT and Gemini services, aiming to develop our own system, titled ReAvaLIA. In our tests, the two services achieved a quadratic weighted kappa score of 0.701 and 0.694, respectively, on a corpus of 200 essays. In this process, ChatGPT had 62% of its evaluations validated in a real exam context, while Gemini achieved 64%, demonstrating that, in a school or course setting, these systems could be used with satisfactory performance.