Customer-service AI is increasingly being judged on its ability to complete tasks independently, yet completion is not always the same as a positive outcome.
Resolution rates can offer a useful snapshot of what an AI system is doing, but they often fail to show whether customers were already frustrated, had contacted the business before or still need support.
To solve this, CX leaders should build measurement frameworks that identify these gaps and use human expertise to improve the automated experience.
Mitch Lieberman, VP of Product and CX at TELUS Digital, told CX Today that a technically resolved query can still leave customers feeling unheard.
“It wouldn’t be a success at the perception of the customer because they don’t feel heard,” he said.
“The customer’s specific transaction may have been solved, but they don’t feel they’ve actually made the case”
Why Your Metrics May Miss Meaning
Today, CX organizations often use resolution rates as an indicator of what an AI system is currently doing.
Whilst this approach provides some necessary insight, they often miss the wider circumstances behind whether a customer actually experienced a successful interaction.
For example, a customer who called several times whilst dealing with an urgent issue is likely to be frustrated by one or more previous interactions.
The resolution rates will record the eventual success from the AI system but not take note of said customer sentiment, resulting in experiences that are never improved.
This is where Lieberman suggests assured access to human support, as a recent Gartner report found that 87% of customers consider access to a human agent essential when companies use GenAI in customer service.
“They have the correct product answer, [but] they may not be aware that I called yesterday or a week ago,” he explained.
“The previous contact history is not all either known to the AI, or if it’s known, it doesn’t know how to interpret that I’ve called last week.”
This requires customer intelligence to look beyond individual interactions and toward repeat contacts, sentiment, and agent interventions to reveal where customers continue to experience friction.
Recency is also relevant as the significance of a request can often change depending on what has happened beforehand.
“Time is a very big part of context awareness,” he said.
“To insert context awareness into a prompt or a request for information from an AI, it doesn’t always know it how to interpret it.”
For CX teams, this means assuring resolution is assessed alongside the signals surrounding each interaction to understand where automation is genuinely supporting the customer.
A human-in-the-loop model can use customer intelligence to identify interactions where emotion or complexity makes immediate human involvement more appropriate.
The Future of Autonomous Service for Human Agents
As the role of autonomous service expands, frontline agents are likely to become increasingly necessary to ensure an AI agent remains within its boundaries and can intervene when interactions require human judgment.
Without removing the importance of human expertise, autonomous service shifts where it’s applied.
Our human customer service agents are the domain experts,” Lieberman said.
“That’s still going to be the case. They’re going to lead AI agents.”
This gives experienced employees responsibility over AI performance instead of handling every conversation themselves.
Introducing dashboards could help identify patterns and individual interactions where automation is struggling, allowing agents to intervene before an issue becomes a wider problem.
Lieberman explained his strategy:
“They may look at a dashboard and say, ‘here are seven AI interactions that are going well, and here’s one that’s not’. I’m going to jump in here and say, ‘I see that there’s a little bit of a struggle in this discussion. Let me see if I can help us work through it.’”
These interventions can also become a source of customer intelligence by highlighting where the system needs refinement, helping to preserve knowledge that might otherwise be lost as experienced employees eventually move away from handling individual contacts.
Designing a strong workforce strategy therefore becomes an important part of AI deployment.
Future agents will need enough decision visibility to understand where an automated response or their own knowledge should take precedence, making institutional expertise part of the feedback loop that improves AI performance.
“The future will be our best domain experts that have the institutional knowledge that we can’t let go, guiding the AI,” he concludes.
Check out the full interview with Mitch Lieberman above to find out more about TELUS Digital’s approach of assessing AI and human interactions, and whether separate AI-performance dashboards obscure the real experience customers receive.
