Study Reveals AI Slows Down Developers by 19%, Contrary to Popular Belief
A recent study conducted by METR has shed light on the unexpected impact of AI on software development productivity. The findings are startling, revealing that AI tools may actually slow developers down, rather than speeding up their work.
The randomized controlled trial involved 16 seasoned open-source developers. It was discovered that when these developers used AI tools, they took 19% longer to complete tasks compared to when they worked without AI assistance.
The study delved into 246 real issues across large repositories. These repositories averaged over 22,000 stars and contained more than a million lines of code. The developers were tasked with genuine bug fixes, feature additions, and refactors that would typically be part of their regular workload.
The results of this study stand in stark contrast to the impressive AI benchmark scores and anecdotal reports of AI’s usefulness. This suggests that while AI may perform exceptionally well in controlled benchmark environments, its effectiveness in real-world deployment scenarios may be quite different.
The research underscores the importance of measuring AI’s actual impact on productivity, rather than relying solely on theoretical capabilities. This study provides a much-needed counterpoint to benchmarks, highlighting the need for a more comprehensive understanding of AI’s role in software development.
Source: METR