AI Implementation · 8 min read
Agent Evals
The Omni model: one managed AI Employee owns one recurring workflow; specialist employees add capacity around the same business context; your team keeps the judgment calls.
A dental practice client called last Tuesday because their AI scheduling agent had approved a same-day appointment for a new patient who, according to the front desk, hadn't been verified as insured. The agent had followed the script. It had pulled the right availability. It had even sent a confirmation. But it skipped a verification step that existed in the workflow for a reason.
The client caught it within three days. We caught it because the eval suite flagged it within hours. Without evals, this kind of failure tends to surface in monthly review meetings, after a handful of patients have already been mishandled. That's the gap we're going to close in this article.
What "Agent Evals" Actually Means
Agent evals are the structured tests you run against an AI agent to determine whether it's doing its job correctly, safely, and consistently. For traditional software, you write unit tests and integration tests. For AI agents, you need an additional layer because the same input can produce different outputs, and "correct" isn't always binary.
We treat evals as three distinct layers:
- Unit evals — isolated prompts and tool calls evaluated against expected behavior. "When a customer asks for a refund over $200, does the agent route to a human?"
- Integration evals — multi-step workflows tested end-to-end with simulated users, real APIs, and realistic edge cases.
- Production evals — ongoing sampling of live conversations, scored against a rubric, reviewed by humans on a schedule.
McKinsey's research on AI deployment consistently finds that companies capturing real value from AI spend significantly more time on testing and iteration than companies stuck in pilot mode. The gap isn't model quality. It's evaluation discipline. Their 2024 state-of-AI report noted that high-performing organizations were roughly twice as likely to have well-defined processes for evaluating AI outputs.
The Four Dimensions We Score On
Most teams start by measuring only whether the agent got the right answer. That's necessary but not sufficient. We score every eval across four dimensions:
- Correctness — Did the agent produce an accurate result? Did it look up the right record, quote the right price, follow the right policy?
- Scope adherence — Did the agent stay inside its lane? If it's a booking agent, did it start offering medical advice?
- Tone and brand fit — Does the response match how the business communicates? A law firm and a pediatric clinic should sound different.
- Escalation judgment — Did the agent know when to hand off to a human? This is the dimension most often underweighted, and it's where most of our production incidents start.
Each dimension gets a 1–5 score from a human reviewer, or an automated rubric where possible. We never use "pass/fail" because real-world agent behavior is rarely binary. What matters is the trend, the failure rate, and whether the agent is improving or drifting.
How We Run Evals at Omni Studio
Here's the workflow we walk every client through. It's not glamorous, but it works.
Pre-deployment: Before an agent touches a real customer, it runs through an eval suite of 200–500 test cases drawn from real conversations. We write these cases with the client's team — they know their edge cases better than we do. A scheduling agent might have 40 eval cases just for "patient wants to reschedule for the third time."
Shadow mode: For one to two weeks, the agent runs alongside the human workflow, observing and suggesting actions without executing them. Every suggestion is scored. This is where you catch the failure modes that don't show up in synthetic test cases.
Production sampling: Once the agent is live, we sample 5–15% of conversations weekly, depending on volume and risk. A high-volume support agent for a SaaS company might be sampled at 2%. A voice agent handling new patient intake for a healthcare practice gets sampled at 25% for the first month.
Human review meeting: Every Friday, a member of our team and a member of the client's team sit down for 30 minutes and review the flagged conversations. Not the wins. The flagged ones. We document patterns, update the eval suite, and adjust the agent's prompts or guardrails.
This is the loop. It doesn't end. As Harvard Business Review has covered extensively, the organizations getting real value from AI treat it as an ongoing operational practice, not a one-time deployment. The eval loop is what makes that operational practice real.
A Real Implementation Scenario
Let me walk through a specific case. We built a voice agent for a multi-location dental practice in the Midwest. The agent's job: answer inbound calls, verify insurance when possible, schedule appointments, and route complex questions to the front desk.
The eval suite had three layers:
- Intent recognition evals — 180 test cases covering accents, background noise, and the specific ways patients ask about insurance ("Do you take Delta Dental?" vs. "Is Delta Dental in-network?").
- Workflow evals — 90 end-to-end test cases simulating full calls, including the verification step that later caused the incident I mentioned at the top.
- Escalation evals — 50 scenarios where the agent should hand off: medical emergencies, billing disputes, new patients without records on file.
Pre-deployment pass rate: 94%. Solid for a voice agent. We deployed to shadow mode.
Day 6 in shadow mode, the eval flagged a pattern: when a caller said "I'm a new patient," the agent was skipping the insurance verification step about 30% of the time. The intent was being classified correctly. The workflow branch was being triggered. But a specific phrase pattern — "new patient" followed by an immediate appointment request — was being short-circuited by a prompt instruction that said "prioritize quick scheduling for new patients."
The fix took 20 minutes. We tightened the prompt, added three new eval cases targeting that exact phrase pattern, and re-ran the suite. Pass rate held at 94% but the failure distribution shifted. The verification step now ran consistently.
Had this gone to production without the eval loop, the practice would have onboarded uninsured patients at scale. The client would have noticed in the monthly billing reconciliation. By then, hundreds of appointments.
Common Failure Modes We See Repeatedly
After deploying agents across roughly 40 service businesses, certain failure patterns show up over and over. Worth flagging them so you can build evals around them from day one.
Prompt drift. The agent was working fine in week one. By week six, performance has degraded. Usually this means the underlying model behavior shifted, the data the agent pulls from changed, or someone updated an upstream system and didn't tell the agent. The eval catches this because the test cases stop passing.
Context window pressure. The agent has access to long conversation history, a knowledge base, and tool outputs. When the context fills up, behavior often degrades in subtle ways. Evals with deliberately long simulated conversations expose this early.
Escalation hesitation. The agent wants to be helpful. It will sometimes answer questions it should be routing. This is the most common production incident and the hardest to catch without specific eval cases. We write at least 20 escalation cases for every agent, even when the client's instinct is that escalation should be "obvious."
Tool calling errors. The agent called the right API with the wrong parameters, or in the right order but at the wrong time. These look like agent failures but are actually integration failures. Evals that mock the API responses cleanly catch this. Evals that rely on live APIs often miss it because the failure gets papered over by a retry.
Gartner's research on AI operationalization has repeatedly emphasized that governance and testing frameworks are the difference between AI projects that scale and projects that stall. The failure modes above are exactly what governance frameworks are designed to catch.
What an Eval Suite Actually Costs to Build
For a typical agent — single workflow, 3–5 intents, two integrations — building the initial eval suite takes us 8–12 hours of work, spread across one to two weeks. That includes writing the test cases, defining the rubric, setting up the scoring infrastructure, and running the first human review cycle.
The ongoing cost is smaller. A weekly review meeting, a few hours of test case maintenance, and a monthly rubric refresh. For most clients, this runs them $1,500–$4,000 per month in our service pricing, depending on volume and complexity. That's the line item that determines whether an agent is a tool you trust or a tool you're constantly worried about.
Frequently Asked Questions
How often should we run evals?
Pre-deployment: every time you change a prompt, a tool, or an integration. Post-deployment: at minimum, weekly production sampling, with the rate scaled up for high-risk workflows. Voice agents and agents handling financial data should be sampled daily for the first 30 days.
What's a "passing" grade on an eval?
We aim for 90%+ across all four dimensions, with zero tolerance for scope violations and escalation failures. A 95% accuracy rate sounds good until you realize that means one in twenty customers is getting a wrong answer. For high-stakes workflows, we push toward 98% before deployment.
Can we run evals in-house without your team?
Yes, but it requires discipline. You need someone who can write test cases, define rubrics, run the scoring infrastructure, and review flagged conversations weekly. Most service businesses we work with don't have that bandwidth internally, which is why they hire us. If you have a strong ops lead, it's possible.
How long does it take to set up an eval suite for a new agent?
About one to two weeks, parallel to the agent build itself. We start writing eval cases during the workflow mapping phase, before the agent is even built. By the time the agent is ready for testing, the eval suite is already in place.
What happens when an eval fails in production?
Three options: rollback the change that caused the regression, tighten the agent's guardrails to prevent the specific failure pattern, or escalate the conversation to a human in real time. We document every production failure in the client's runbook so the team knows how to respond without waiting on us.
Closing Thoughts
An agent without evals is a liability. An agent with a disciplined eval loop is infrastructure you can build on. The difference isn't the model or the prompt — it's the operational practice around it. If your team has deployed an agent without a clear eval framework, that's the first thing we'd look at in an audit.
Want us to review your existing agent setup or map out an eval framework for one you're planning to deploy? Book a free AI automation audit and we'll walk through your current workflow, flag the gaps, and send you a written summary within a week.
