Evaluating accessibility in native Android interfaces generated by Large Language Models

Abstract

Recent advances in artificial intelligence, particularly in large language models (LLMs), have opened up new possibilities for automating software development tasks, including code generation for mobile applications. This study explores the capabilities of LLMs, such as ChatGPT, in improving the accessibility of native Android applications. It examines whether LLM-generated code conforms to established accessibility standards, taking into account different screen layouts, prompt formulations, and interface generation strategies. Four studies were conducted to evaluate the accessibility of seven types of mobile interfaces. The first study analyzed the accessibility of mobile application screens created using a variety of layout strategies. In contrast, the second study focused specifically on Jetpack Compose and compared the output of several LLMs. The third study examined whether creating screens with English prompts affected accessibility. Finally, the fourth study used an LLM code assistant. A total of 702 accessibility issues were identified in all studies. Jetpack Compose consistently outperformed other layout approaches, and English prompts resulted in fewer issues. Interestingly, prompts that explicitly requested accessibility often resulted in more errors, suggesting that LLMs face challenges in correctly interpreting and implementing accessibility requirements. These findings highlight the importance of refining prompt strategies and LLM outputs to reduce the risk of accessibility errors in AI-generated mobile app code.