Avaliando o desempenho de grandes modelos de linguagem na realização de provas de dedução natural em lógica proposicional e lógica de predicados

Abstract

The growing use of conversational agents has sparked increasing interest among researchers, educators, and educational institutions worldwide. Moreover, with the recent popularization of Large Language Models (LLMs), numerous studies have been conducted to explore the use of these tools to support the teaching and learning process. These systems are capable of understanding and processing vast amounts of data, which allows them, for example, to provide individualized support to students in solving exercises. However, it is essential to consider that such systems generally employ reasoning processes based on statistical methods, such as machine learning algorithms, which may produce incorrect answers in tasks requiring logical reasoning. This study aims to evaluate the ability of Large Language Models (LLMs) to solve Natural Deduction exercises in Propositional Logic and Predicate Logic. To this end, experiments were carried out using the GPT 4.1-mini, GPT 4o, and GPT 3.5turbo models to solve natural deduction exercises. These experiments were conducted a priori, without training, and subsequently, further experiments were performed on the trained GPT 4.1-mini model to assess potential performance improvements. The results indicate a significant improvement after training, although the model still exhibits a considerable number of errors. Therefore, if employed for this task, its use is recommended with due caution.