The industry has been grading voice agents all wrong. Accuracy and conversational experience aren’t separate report cards — they’re locked in a tug-of-war, and pulling on one rope slackens the other. That’s the central finding from EVA, a new open-source evaluation framework that tests voice agents the way people actually use them: in chaotic, multi-turn phone calls where misunderstandings pile up and latency kills momentum.
Built by researchers from Daily and released with an initial dataset of 50 airline scenarios — think rebookings, voucher disputes, cancellations — EVA forces agents through bot-to-bot audio conversations that mimic real callers. It then spits out two scores: EVA-A for task accuracy and EVA-X for conversational experience. The team benchmarked 20 different systems, spanning traditional cascade setups (STT → LLM → TTS) and newer audio-native models. The result wasn’t a winner-takes-all. It was a scatter plot.
Agents that actually got your flight changed tended to be clunky conversationalists. Meanwhile, the smooth-talking agents that never interrupted you and kept things brief? They frequently botched the task entirely. One researcher put it bluntly: frameworks that only measure component parts miss the “interaction dynamics that are not apparent at the component level” — like whether a 300-millisecond delay makes a user repeat their credit card number, dooming the whole transaction.
EVA’s public release on GitHub and Hugging Face means any team can now run their agent against the same gauntlet. The framework uses Pipecat, an open-source Python library for real-time voice apps, and deterministic tool execution so results are reproducible. This isn’t just a benchmark; it’s a design constraint made visible. For product teams betting on voice, the message is clear: you’re not building two things. You’re building one thing with two masters, and right now, nobody’s figured out how to please them both.
The airline dataset is only the first domain planned. If the pattern holds — and I suspect it will — we’ll see this same trade-off surface in healthcare scheduling, food ordering, and anywhere a voice agent has to actually do something. The question isn’t whether to optimize for accuracy or experience. It’s how to acknowledge you’re sacrificing one for the other, and whether your users will tolerate the one you chose to neglect.
