The new performance metric for AI-assisted software development

When organizations evaluate the success of their generative AI investments, the dashboard metrics often appear encouraging. Global benchmarks reinforce this optimism: 84% of developers are actively integrating AI assistants into their workflows, and GitHub reports that more than 80% of new developers use CoPilot in their first week.

For CIOs and business leaders, the conclusion may seem straightforward:

However, measuring success by adoption metrics alone creates a significant strategic blind spot.

While generative AI can significantly accelerate individual developer tasks, translating those gains into measurable improvements across the software development lifecycle is far more complex.

This raises a fundamental question for engineering leaders:

If AI tool adoption is approaching ubiquity, why aren’t engineering outcomes improving at the same pace?

The answer lies in what organizations choose to measure.

Most organizations continue to track the following metrics:

• Number of AI licenses
• Active users
• Prompt volumes
• Frequency of AI usage

These indicators provide visibility into tool adoption, not developer capability.

Organizations today have increasingly sophisticated mechanisms for monitoring AI adoption, yet comparatively little insight into how effectively developers collaborate with AI to produce secure, maintainable, and high-quality software.

Providing developers with access to powerful AI tools can significantly accelerate code generation. However, generating more code does not always lead to better software.

Without a structured approach to AI-enabled development and a clear framework for effective AI usage, organizations risk creating technical debt that makes software harder to maintain over time, even when it is built faster.

These findings show that while AI can accelerate development, speed alone is not a reliable indicator of engineering quality. Without appropriate oversight and sound engineering judgment, faster development can come at the expense of maintainability, security, and long-term resilience.

Software engineering is shifting from code production to code governance.

Consider two developers using the same AI tool.

One accepts AI-generated recommendations without validation, introducing the likelihood of logical defects, security risks, and unnecessary technical debt.

The other gives clear context, critically evaluates AI outputs , validates recommendations, and maintains architectural quality throughout the development process.

While both developers use the same AI tool, the key difference is not access, but in the ability to apply AI effectively.

Traditional programming capability remains indispensable. However, AI-assisted development adds a new dimension of performance, applied AI capabilities, that conventional technical assessments fail to measure.

The critical question for engineering leaders is:

How effectively are developers applying AI in real software engineering workflows?

Answering this requires objective data that enables organizations to:

• Identify gaps in AI capability
• Design targeted upskilling programs
• Reduce engineering and architectural risks
• Prepare for more autonomous AI-enabled development environments

To help organizations evaluate these emerging capabilities, Mercer’s AI Agility (developer version) assessment was built to measure the applied skills required for effective software development in AI-enabled environments.

The assessment measures five core capability areas:

• LLM foundations and concepts – Understanding how large language models work and their practical capabilities and limitations.
• Prompt engineering – Communicating effectively with AI to generate relevant, accurate, and context-aware outputs.
• Code verification and quality – Reviewing AI-generated code, identifying logical issues, validating correctness, and improving solution quality.
• Security hygiene and compliance – Applying AI responsibly while protecting sensitive data, ensuring secure coding practices, and understanding AI-related risks.
• Business context and ROI – Making informed decisions about when AI adds value, balancing productivity with engineering quality and business objectives.

Together, these capabilities offer a structured view of the applied skills needed to succeed in AI-enabled software development teams.

The evolution of software development is still unfolding. As organizations move from AI-assisted coding to AI-enabled workflows and eventually towards more autonomous software engineering practices, the expectations placed on developers will continue to evolve.

Success will depend not only on technical proficiency, but also on the ability to collaborate effectively with AI, exercise sound judgment, validate AI-generated outputs, and use AI responsibly within complex engineering environments.

In this changing landscape,