Uma investigação empírica da engenharia de prompts em LLMs como estratégia para a geração de histórias de usuário

Abstract

Large language models (LLMss) have been applied to support Requirements Engineering activities, such as drafting user stories. However, uncertainty remains regarding the stability of these generations when the same process is repeated and how such stability relates to the quality perceived by specialists. This study investigated the semantic consistency of user stories generated by three LLMss (ChatGPT 5 Instant, DeepSeek, and Gemini 2.5 Pro) across four domains by repeating a protocol based on the 3Cs. Consistency was estimated using text embeddings and cosine similarity, considering only stories that were comparable across repetitions (i.e., produced under the same user story identifier). In parallel, a controlled sample was assessed by two specialists in two domains using criteria inspired by the Quality User Story framework (QUS). The results indicated moderate to high consistency, varying by model and domain: ChatGPT and DeepSeek were more predictable, whereas Gemini behaved more exploratorily, generating a larger number of stories but showing less uniform repeatability across runs. In the human evaluation, the stories were deemed useful as a starting point, yet recurring weaknesses were observed in atomicity and testability. Overall, semantic consistency and perceived quality capture different properties and should be treated as complementary measures.