Avaliação empírica da eficácia de Modelos de Linguagem Grande (LLMs) na refatoração de Projetos Python

Abstract

Code refactoring is an essential practice to ensure the quality and continuous evolution of software systems, especially in languages like Python, which demand high maintainability. Although static analysis tools such as SonarQube provide support in identifying issues, the refactoring process still presents challenges, such as preserving functionality and improving code readability. In this context, Large Language Models (LLMs), such as GPT-4, DeepSeek, and Claude AI, emerge as promising tools by combining advanced contextual analysis with automated code generation. This study aims to evaluate the effectiveness of LLMs in refactoring Python code, focusing on correcting maintainability issues, identifying limitations, and proposing improvements. To this end, we conducted an empirical study with four widely used models: Copilot Chat 4o, LLaMA 3.3 70B Instruct, DeepSeek V3, and Gemini 2.5 Pro. Furthermore, this study also investigates whether these models perform better when used with more refined prompting techniques, such as few-shot learning. For this purpose, each LLM was submitted to two distinct prompting styles: zero-shot and few-shot, allowing a comparative analysis of the impact of these approaches on the quality of the generated refactorings. We evaluated 150 methods with maintainability problems by LLM and by prompt technique, and the results indicate that, although the models achieved considerable effectiveness rates in the few-shot scenario—Gemini (64.67%), DeepSeek (64.00%), Copilot (63.33%), and LLaMA 3.3 70B (55.33%), all faced significant limitations. Among the main challenges observed were the introduction of new maintainability problems, runtime errors, failures in automated tests, and, in some cases, the failure to correct the originally identified issue. Additionally, we conducted a human participant evaluation to analyze the readability of the code refactored by the models. The results indicate that 81.25% of the solutions were perceived as improvements, especially in structural aspects. However, there were also cases where readability was impaired, either due to the introduction of unnecessary complexity or lack of standardization in code style. These findings reinforce the need for caution when automatically adopting suggestions generated by LLMs, and highlight the importance of validation by developers in the final code review. This work contributes a comparative analysis of the capabilities of LLMs, pointing out their limitations and proposing practical methodologies for integrating AI into the code refactoring process. The results of this study aim to pave the way for new research, especially in the development of more efficient prompting techniques and in the evaluation of models yet to come. We hope these contributions help developers and researchers find more practical, reliable, and long-lasting solutions to improve software maintainability in everyday practice.