Customer-service AI is increasingly being judged on its ability to complete tasks independently, yet completion is not always the same as a positive outcome.
Resolution rates can offer a useful snapshot of what an AI system is doing, but they often fail to show whether customers were already frustrated, had contacted the business before or still need support.
To solve this, CX leaders should build measurement frameworks that identify these gaps and use human expertise to improve the automated experience.
Mitch Lieberman, VP of Product and CX at TELUS Digital, told CX Today that a technically resolved query can still leave customers feeling unheard.
“It wouldn't be a success at the perception of the customer because they don't feel heard," he said.
"The customer's specific transaction may have been solved, but they don't feel they've actually made the case”
Why Your Metrics May Miss Meaning
Today, CX organizations often use resolution rates as an indicator of what an AI system is currently doing.
Whilst this approach provides some necessary insight, they often miss the wider circumstances behind whether a customer actually experienced a successful interaction.
For example, a customer who called several times whilst dealing with an urgent issue is likely to be frustrated by one or more previous interactions.
The resolution rates will record the eventual success from the AI system but not take note of said customer sentiment, resulting in experiences that are never improved.
This is where Lieberman suggests assured access to human support, as a recent Gartner report found that 87% of customers consider access to a human agent essential when companies use GenAI in customer service.
“They have the correct product answer, [but] they may not be aware that I called yesterday or a week ago," he explained.
"The previous contact history is not all either known to the AI, or if it's known, it doesn't know how to interpret that I've called last week.”
This requires customer intelligence to look beyond individual interactions and toward repeat contacts, sentiment, and agent interventions to reveal where customers continue to experience friction.
Recency is also relevant as the significance of a request can often change depending on what has happened beforehand.
“Time is a very big part of context awareness," he said.
