Um benchmark para avaliação de grandes modelos de linguagem no português usando questões do exame nacional do ensino médio

Abstract

Large Language Models (LLMs) have become fundamental to many applications, including artificial intelligence, natural language processing and automation systems. However, evaluating their performance in Portuguese still presents challenges, since most of the available benchmarks are focused on English. This paper proposes a benchmark for evaluating LLMs in Portuguese, using multiple-choice questions taken from the National High School Exam (ENEM) from different years. The aim is to measure the accuracy of the models and their adherence to the explicit instructions in the prompts. To this end, several open source models were tested, evaluating their accuracy and compliance in the formatting of the answers. The results indicate that there are significant variations in the performance of the models, highlighting the need to improve their adaptation into Portuguese. In addition, the study contributes to the creation of more effective methodologies for evaluating and developing LLMs for the Portuguese language.