Avaliação comparativa de modelos de IA especializados na geração de código: um estudo com LLMs populares da Hugging Face

Abstract

This study presents a comparative evaluation of LLMs specialized in code generation, available on the Hugging Face platform: codegen-350M-mono, Qwen2.5-Coder-0.5B-Instruct, deepseekcoder-1.3b-base, tiny_starcoder_py, and replit-code-v1-3b. Using the lm-evaluation-harness tool, the models were assessed through standardized benchmarks, considering metrics such as pass@1, acc, json_validity, and schema_compliance, aiming to measure their effectiveness in functional programming tasks and structured data generation. The methodology involved automated execution in a controlled environment (Google Colab), result collection, and statistical analysis. Experimental results showed that the replit-code-v1-3b model achieved the best overall performance, followed by deepseek-coder-1.3b-base, particularly in tasks requiring semantic accuracy and compliance with JSON structures. The study concludes that more robust models, fine-tuned for specific tasks, show greater applicability in real-world software development scenarios, indicating they should be preferred in environments demanding high quality and reliability.