The question of how to measure AI in customer experience is becoming harder to sidestep. As more CX teams build out their AI deployments, the pressure to show real ROI is sharpening, and a growing number of leaders are realizing that the metrics they’ve relied on simply aren’t telling the full story.
In a recent conversation with CX Today, Brad Birnbaum, CEO and Co-Founder of Kustomer, made the case that the industry is stuck in the wrong measurement mindset. Deflection rates and average handle time still have a place, he argues, but they were never designed to capture business outcomes, and that gap is becoming a problem:
“It’s not about having AI do something for you. It’s about getting the value of that something, getting the outcome of that something.”
The conversation covers how that shift plays out in practice. On metrics, Birnbaum is deliberately un-prescriptive: the right outcomes depend entirely on the business.
A retailer might focus on turning a return into an exchange and protecting revenue. A subscription business might prioritize churn prevention. A B2B SaaS company might track upsell activity. The point is defining your North Star first, then letting AI work toward it.
The discussion also addresses AI observability, a topic drawing increasing attention from CX leaders who want to understand not just what AI is doing, but why.
Birnbaum describes Kustomer’s evaluation framework, which runs continuous iterations to tune AI responses, and the platform’s governance tools that keep interactions within defined boundaries.
On the human-in-the-loop question, he outlines a fluid model where AI and agents hand off to each other based on the complexity and stakes of an interaction, with Kustomer’s stated goal being to shift human agents from creators to reviewers.
For leaders figuring out where to start, his advice is consistent: pick your most important outcomes, get something live quickly, and build from there.
