English
The new performance metric for AI-assisted software development
When organizations evaluate the success of their generative AI investments, the dashboard metrics often appear encouraging. Global bench,marks reinforce this optimism: 84% of developers are actively integrating AI assistants into their workflows [1], and GitHub reports that more than 80% of new developers use CoPilot in their first week [2].
For CIOs and business leaders, the conclusion may seem straightforward:
The engineering workforce has embraced AI, and development velocity should naturally improve.
However, measuring success by adoption metrics alone creates a significant strategic blind spot.
While generative AI can significantly accelerate individual developer tasks, translating those gains into measurable improvements across the software development lifecycle is far more complex.
Increased productivity at the task level does not automatically lead to better engineering outcomes.
This raises a fundamental question for engineering leaders:
If AI tool adoption is approaching ubiquity, why aren’t engineering outcomes improving at the same pace?
The answer lies in what organizations choose to measure.
Most organizations continue to track the following metrics:
- Number of AI licenses
- Active users
- Prompt volumes
- Frequency of AI usage
These indicators provide visibility into tool adoption, not developer capability.
Organizations today have increasingly sophisticated mechanisms for monitoring AI adoption, yet comparatively little insight into how effectively developers collaborate with AI to produce secure, maintainable, and high-
The true cost of unguided AI adoption
Providing developers with access to powerful AI tools can significantly accelerate code generation. However, generating more code does not always lead to better software.
Without a structured approach to AI-enabled development and a clear framework for effective AI usage, organizations risk creating technical debt that makes software harder to maintain over time, even when it is built faster.
A multi-year GitClear analysis of over 600 million lines of code revealed several trends that explain the growing maintainability challenge:
These findings show that while AI can accelerate development, speed alone is not a reliable indicator of engineering quality. Without appropriate oversight and sound engineering judgment, faster development can come at the expense of maintainability, security, and long-term resilience.
The core competency of the AI-augmented engineer
Software engineering is shifting from code production to code governance.
Consider two developers using the same AI tool.
One accepts AI-generated recommendations without validation, introducing the likelihood of logical defects, security risks, and unnecessary technical debt.
The other gives clear context, critically evaluates AI outputs , validates recommendations, and maintains architectural quality throughout the development process.
While both developers use the same AI tool, the key difference is not access, but in the ability to apply AI effectively.
As AI capabilities continue to evolve, developers are increasingly expected to:
Closing the gap
The critical question for engineering leaders is:
How effectively are developers applying AI in real software engineering workflows?
Answering this requires objective data that enables organizations to:
- Identify gaps in AI capability
- Design targeted upskilling programs
- Reduce engineering and architectural risks
- Prepare for more autonomous AI-enabled development environments
Introducing AI agility (developer version)
To help organizations evaluate these emerging capabilities, Mercer’s AI Agility (developer version) assessment was built to measure the applied skills required for effective software development in AI-enabled environments.
The assessment measures five core capability areas:
- LLM foundations and concepts – Understanding how large language models work and their practical capabilities and limitations.
- Prompt engineering – Communicating effectively with AI to generate relevant, accurate, and context-aware outputs.
- Code verification and quality – Reviewing AI-generated code, identifying logical issues, validating correctness, and improving solution quality.
- Security hygiene and compliance – Applying AI responsibly while protecting sensitive data, ensuring secure coding practices, and understanding AI-related risks.
- Business context and ROI – Making informed decisions about when AI adds value, balancing productivity with engineering quality and business objectives.
Together, these capabilities offer a structured view of the applied skills needed to succeed in AI-enabled software development teams.
Looking ahead
The evolution of software development is still unfolding. As organizations move from AI-assisted coding to AI-enabled workflows and eventually towards more autonomous software engineering practices, the expectations placed on developers will continue to evolve.
Success will depend not only on technical proficiency, but also on the ability to collaborate effectively with AI, exercise sound judgment, validate AI-generated outputs, and use AI responsibly within complex engineering environments.
In this changing landscape,
References
Get awesome marketing content related to Hiring & L&D in your inbox each week
Stay up-to-date with the latest marketing, sales, and service tips and news
By using our offerings and services, you are agreeing to the Terms of Services and License Agreement and understand that your use and access will be subject to the terms and conditions and Privacy Notice