Modelos de linguagem Transformer: uma análise comparativa de abordagens de construção, especialização e otimização

Abstract

This work undertakes a systematic evaluation of three distinct approaches for developing Transformer-based language models, aiming to optimize the trade-off between performance and computational cost. The first approach consists of constructing a model from its fundamental components, establishing a foundational understanding of the architecture’s mechanics. The second investigates the specialization of a pre-trained model (GPT-2) through fine-tuning for specific tasks, namely text classification and instruction following. The third approach proposes and validates an optimized training pipeline for a Small Language Model (SLM) with 15 million parameters, leveraging high-quality data and computational efficiency techniques. The findings demonstrate the effectiveness of fine-tuning as a powerful method for adapting models to specialized applications. Critically, the research validates the hypothesis that high-quality data can compensate for a reduced model size, enabling the SLM to achieve remarkable coherence and grammatical correctness, challenging the paradigm that larger models are invariably superior. The study concludes that the development of effective models depends on a strategic balance between model scale, data quality, and available computational resources, highlighting SLMs as a viable and sustainable alternative.