Desenvolvimento de um Largue Language Model: uma estratégia de aprimoramento de transcrições de anamneses médicas

Abstract

This work aims to explore the development of a Large Language Model (LLM) to enhance the transcription of patient records. The research involved two main stages: the first consisted of an initial benchmark of three language models (Phi3, Llama, and Mistral), evaluating their performance in the task of formatting anamneses, and the second focused on fine-tuning the Mistral model, based on qualitative and quantitative feedback from the evaluators. The Mistral model stood out in the benchmark due to its consistency and better results in agreement metrics, such as Kendall’s tau and Fleiss’ Kappa coefficients, although it was criticized for overloading details in its responses, impairing its objectivity. The Llama model performed the worst, with responses that were often irrelevant, while Phi3 showed moderate performance, but with issues in precision and excessive use of generic language. The fine-tuning of the Mistral model, despite attempts to correct the flaws, did not result in significant improvements, as the average evaluation score was slightly lower than the original model. The main contributions of this work include the comparison of language models for a specific task in the medical field and the analysis of the limitations of the fine-tuning process. The study also identified important limitations, such as data quality and potential bias among the evaluators, which may have influenced the results. As future work, the intention is to conduct longer training sessions, expand and diversify the dataset, and explore new machine learning approaches to further improve the model’s performance.