How we size voice agents for enterprise traffic
The load testing tool we built, what the measurements showed, a scheduling bug that took us too long to find, and the Erlang model we now use for sizing.
When a customer asks how many calls our voice agent can handle, the honest answer used to be "it depends". This post is about how we turned that into a number we can defend. It covers the load testing tool we built, what the measurements showed, a scheduling bug that took us too long to find, and the Erlang model we now use for sizing.
The target was about 1 million calls a month during working hours, which works out to roughly 812 calls in progress at once based on Erlang.
Why requests per second doesn't work
For a normal web service you measure requests per second and you're mostly done. A voice call is different. It stays open for minutes, and during that time the agent is listening, detecting when the caller has stopped talking, calling a model, and generating speech. How much work that is depends on the conversation. A caller who talks in short bursts costs a lot more than one who pauses a lot, even if both calls last the same time.
So call volume alone doesn't tell you how much hardware you need. We had to measure it.
The stack we're testing
Everything in the voice pipeline is self-hosted. No part of a call goes to a third-party API, so a load test measures the whole system we run, and every part of it is something we can tune.
- Speech to text is a model we fine-tuned on our own labelled call centre recordings. Call centre audio is narrowband and noisy, and callers often mix languages in one sentence. General purpose models struggle with that.
- Turn detection uses Scicom Semantic VAD. A plain voice activity detector decides the caller has finished when they go quiet for long enough, so it cuts in whenever someone pauses to think. Scicom Semantic VAD listens to the last few seconds of audio and predicts whether the caller has actually finished or has only paused mid-sentence. It works on audio alone and doesn't need a transcript. In our benchmarks it cut callers off about a third less often than silence-based detection, at the same response latency. It runs on a GPU. If you want to run turn detection on CPU, we've open sourced smaller models trained the same way: whisper-tiny, whisper-base and whisper-small.
- The language model is served with NVIDIA Dynamo using disaggregated serving. Prefill, which processes the prompt, and decode, which generates tokens, run on separate workers. A long prompt arriving for one call doesn't slow down token generation for the others, which matters when every turn is latency sensitive.
- Text to speech is our own Multilingual Expressive TTS model. It streams audio, so the agent starts speaking before the whole reply is synthesised.
- Media runs through a self-hosted WebRTC media server.
The load testing tool
We built an internal tool that load tests voice agents end to end. It places real calls over the same media path a customer would use. Each simulated caller has a persona and a scenario. It speaks with synthesised audio, waits for the agent to reply, and hangs up when the scenario ends. Every call produces a transcript, a recording, latency per turn, and a pass or fail result. A load test runs many of these at the same time and holds the level for a fixed period.
It isn't limited to one kind of agent. The same test definition can run against:
- WebRTC voice agents, joining the same media rooms a browser or app user would.
- Phone agents over SIP, placing a real call through a carrier trunk, so the results include the telephone network's own latency and audio quality.
- Chat agents, over a plain JSON webhook or a streaming event protocol, where the measure is how long the reply takes.
Personas and scenarios are shared across all three, so one conversation design can test a phone line, a web widget and a chat window. Callers can also be given different real voices, with different accents and background noise, so the agent isn't only ever tested against one clean synthetic voice.
We built it for enterprise-scale runs. Every simulated call is its own isolated worker, so a test scales out by adding workers rather than by pushing one process harder. A ramp can go straight to a level, climb in steps, or hold for a long soak. Each session keeps its own transcript, recording and trace, so any single call in a large run can be opened and replayed. Thresholds are set per pipeline stage (speech recognition, the model, speech synthesis, end-to-end response), and a breach is reported as it happens rather than stopping the test, so you still see where the curve goes past the limit.
A few implementation details ended up mattering a lot:
- We start the latency clock when the caller's audio has finished playing on the far side, not when our code finished sending it. The media stack buffers audio, so the send returns early. Measuring from the send made every agent look faster than it was.
- We use the agent's own transcript of what it said instead of transcribing its audio ourselves. Our transcription only finishes after a silence window, so it arrives late. We label every latency with the source it came from and never mix the two.
- If the caller's speech recognition service fails, the test keeps going and marks the result as degraded. In an early version this was fatal, and one bad response from a shared service wiped out a whole run.
Agents fail by dropping conversations
We expected the usual pattern: as load goes up, latency goes up, and eventually things get slow. That's not what happened.
- Conversations that finish
Below a certain load, every call is answered and every conversation finishes. Above it, conversations start failing partway through. The caller is still on the line but the agent stops keeping up, and the call times out.
This happens before the CPU is fully used. The agent framework has admission control: each worker checks how responsive it is, and when that drops below a threshold it stops taking new calls. So the agent starts protecting itself earlier than you'd expect from looking at CPU.
This is why we measure completion instead of connection. A call that connects and then fails after two exchanges counts as answered, but it's a failure for the customer. We only count a call as successful if the conversation reaches its planned end.
After we fixed the problem described later in this post, this is how calls ended at each load level:
- Completed
- Died mid conversation
- Never answered
Almost no calls fail midway. Once the agent is at capacity it rejects new calls instead of accepting them and dropping them later, which is the behaviour we want. We could only see this because we counted rejected calls and dropped calls separately.
Two SLOs
To define capacity we needed two thresholds: a minimum completion rate and a maximum response latency.
- Conversations that finish
- Answer latency, tail
They break at different load levels. In our tests completion broke first, with a clear gap before latency did. Inside that gap the latency numbers look fine while some conversations are already failing. If you only track latency you'd call that range healthy. We define capacity as the load where the first of the two thresholds is crossed.
The median doesn't move
- Fleet doubled
- Doubled again, piled onto one host
- Doubled again, spread across hosts
Solid lines are the slowest one in twenty replies; dashed lines are the median for the fleet of the same colour.
The dashed lines are the median time to first response for each fleet configuration. They're almost flat across the whole range we tested. The solid lines are the slowest replies, and those are what change with load.
The median is flat because it's mostly fixed pipeline cost: speech recognition finishing, the model producing its first token, and speech synthesis producing its first audio. These happen one after another on every turn, and more hardware doesn't make them faster. It only shortens queues, and at the median there isn't much of a queue.
This limits what we can promise. If someone asks for a latency target below that floor, more servers won't get there. The work has to happen in the pipeline itself: faster recognition, a faster first token, or streaming synthesis. We now state the floor at the start of every capacity discussion.
Joining test results with Prometheus
Test results show when something breaks but not why. So for each load test we query Prometheus for the same time window and pull CPU, memory, throttling and host utilisation for the agent pods, the media server, and the load generator.
We include the load generator because if it runs out of resources, the whole test is measuring the wrong thing.
Two design choices:
- Metrics are read from Prometheus when you view a test, not copied into the test result. Prometheus already stores them, and a second copy can drift. It also meant that when we added the feature, it worked on tests we had already run.
- Users configure which Prometheus to use and which namespace and deployment the agent runs in. The queries themselves live in code, so a mistake shows up as a failing test instead of an empty chart.
- Agent process
- Load generator
- Media server
Each component here is plotted against its own CPU limit. The agent pods climb with load and level off near the point where they start rejecting calls. The media server and the load generator barely move. The media server only forwards audio packets. It doesn't decode or re-encode audio or run any models, so it costs far less per call than the agent, and it was never throttled in our tests. We had worried about the media layer, and the data showed the limit was somewhere else.
How many LLM calls a turn makes
One turn looks like one LLM call from the agent's side, but our agent is agentic. Before it answers it routes the request, picks tools, and checks its own output, and each of those is a separate call to the model. The agent's own telemetry only reports one completion per turn: the final reply the caller hears. The rest don't show up anywhere on the agent side.
So we counted at the model server instead. The language model runs on vLLM behind NVIDIA Dynamo, and it exports its own request counter to Prometheus. We read that counter for the same load test windows and compared it with the number of agent turns in each window.
Two things had to come out first. Our simulated caller uses the same model through the same gateway, one call per caller turn, so we subtracted those. Other teams also use the same model, so we didn't subtract an idle baseline. Instead we fitted a line through all the load levels. The slope is the calls per turn, and the intercept absorbs the other traffic.
The result was several LLM requests per turn. Only one of them is the reply; the rest are agentic calls the agent never reports. Speech to text and text to speech stay at one request per turn each.
- Counted at the model server
- Reported by the agent
This matters for sizing the model servers. If we had sized them on the agent's own count, we would have under-provisioned them several times over.
In our tests the model was never the bottleneck. Requests to it climbed with load, but its response time stayed flat and almost nothing queued at any load level.
- Requests per second
- Time to first token, tail
- Requests waiting in queue
Each line is on its own scale; the chart compares their shapes, not their values.
But our tests only reached a small fraction of the request rate our target needs, so the model servers need their own load test at that rate.
The scheduling bug
For a while the agent performed much worse than it should have. The confusing part was that adding replicas made it worse. Doubling the fleet helped. Doubling it again made completion drop below what the smaller fleet had managed.
- Baseline fleet
- Fleet doubled
- Doubled again, piled onto one host
- Doubled again, spread across hosts
Throughput had the same problem. The larger fleet peaked at a low load and then did less work as we added more calls.
- Baseline fleet
- Fleet doubled
- Doubled again, piled onto one host
- Doubled again, spread across hosts
The rejected calls metric didn't help either. The smaller fleets rejected calls when they were full, which is easy to spot. The larger fleet accepted almost every call and then failed during the conversation, which that metric doesn't count.
- Baseline fleet
- Fleet doubled
- Doubled again, piled onto one host
- Doubled again, spread across hosts
At moderate load only about a third of conversations completed, and the slowest replies were many times slower than the median. The first thing we checked was container CPU throttling, to see if the containers were hitting their CPU limit. Throttling was close to zero the whole time.
- Busiest host, before the fix
- Container throttling, before
- Container throttling, after
We spent a long time on this because the metric said the containers had spare CPU, and they clearly didn't.
The cause was pod placement. Most of the replicas had been scheduled onto one host while other hosts were idle. That host was at full CPU, and the replicas on it were starved.
Container throttling can't detect this. It counts the periods where a container uses up its own CPU quota. A container on an overloaded host can't get enough CPU to use up its quota, so it shows almost no throttling while it waits. That waiting shows up as run queue delay, which this metric doesn't measure. Our containers never reached their CPU limit during this period.
We changed the deployment to spread replicas one per host. Nothing else changed: same image, same code, same test. Completion went from about a third to over ninety percent at the same load. Throughput almost tripled. The slowest replies got more than ten times faster. That's the green line in the charts above.
Throttling went up after the fix, by a lot. That's expected, because the containers could now actually reach their CPU limits.
Conversations also started reaching their natural end again:
- Baseline fleet
- Fleet doubled
- Doubled again, piled onto one host
- Doubled again, spread across hosts
What we changed afterwards:
- We always look at host utilisation next to container throttling.
- We set pod placement explicitly instead of relying on the scheduler's defaults. The scheduler did what our config told it to. The config was too narrow and nothing warned us.
Bugs in our own measurements
Two bugs in our own tooling made it into conclusions before we caught them.
The first was the CPU limit. Our cluster has two scrape jobs that both publish the same Kubernetes metrics, so summing the CPU limit counts it twice. We fixed that by removing duplicates before summing. But an older cached copy of the result was still feeding our report, and it was wrong in a different way. Every utilisation percentage in the report was half the real value. There was a sign we should have caught sooner: the report had manual correction factors in the few places where someone had noticed the numbers looked off. Now these values come from one query with duplicates removed, and we don't cache derived values.
The second was a memory estimate. We fitted memory usage against load for a fleet of fixed size. The fit's intercept was the baseline for the whole fleet, not for one replica. When we used it to size a much larger fleet, we scaled the slope but kept the intercept, and the memory estimate came out at less than half the real figure. We caught it because we had two implementations of the sizing model, one in Python and one in the browser, and their results didn't match. We now compare the two on every change.
Sizing on turns, not calls
To turn the measurements into a sizing model we needed a unit. Our first choice was calls per replica, and it was wrong.
CPU usage follows the number of conversational turns, not how long the call lasts. Two calls of the same length can use very different amounts of CPU depending on how much back and forth happens.
The scenario we tested with was on the slow side. The real traffic we were sizing for had roughly three times as many turns per minute. If we had sized on calls per replica, we would have under-provisioned by about the same factor.
So the model uses turns per minute: the turns the fleet has to handle, divided by the turns one replica can handle while staying within both SLOs. That holds across different kinds of traffic. The calculator also shows the turn rate we tested at next to its answer, and if someone's traffic is busier, it tells them to re-test.
Monthly volume isn't concurrency
The business question is usually "we expect this many calls a month, can we handle it?" A monthly call count doesn't tell you how many calls are active at once.
How many calls are in progress at any moment depends on how long each call lasts. The same monthly volume gives very different concurrency for one minute calls than for fifteen minute calls.
This is the problem Erlang formulas were created for in telephone networks. Offered traffic, measured in Erlangs, is the arrival rate multiplied by the average call duration. It's the average number of calls in progress.
For our target it works out like this:
- About 1,000,000 calls a month, over 22 working days of 14 hours, is about 3,247 calls an hour.
- We add a 20% buffer for peaks the monthly average hides, which gives about 3,896 calls an hour.
- The agent handles the whole call, and the average call lasts 750 seconds.
- 3,896 calls an hour × 750 seconds ÷ 3,600 seconds per hour = about 812 calls in progress at once.
Call length matters as much as volume here. With a one minute average call, the same monthly volume is only about 65 calls at once.
- Chance a caller is turned away
Calls don't arrive evenly, so the number in progress is above the average about half the time. If you provision exactly the average, you reject a lot of calls. The Erlang B formula tells you how many lines you need to keep the rejection rate under a target. The extra capacity above the average is there because arrivals are random, and it's often missing from spreadsheet capacity plans.
One implementation note: the textbook form of Erlang B uses factorials, which overflow a float well within the range we need. The recursive form doesn't have that problem.
Running the model backwards
The forward question is "we expect this much traffic, what do we need?" Often the question comes the other way: "we have this capacity, how much traffic can we accept?"
Every step can be reversed. Erlang B increases steadily with traffic, so you can binary search for the traffic a given number of lines supports at your target rejection rate, then convert that to a monthly volume.
We made two mistakes here. First, if you add a peak buffer to the arrival rate going forward, you have to remove it going backward, or you count the headroom twice. Second, rounding. The reverse calculation lands exactly on the rejection threshold, so rounding the monthly volume up by a fraction of a call made the forward calculation return one extra line. We found both because we check that the forward and reverse directions agree.
Summary
The calculator takes monthly call volume, working hours, a peak buffer and a call profile, and returns CPU, memory, host count and expected response latency for the SLOs you set. It also works in reverse from a fixed capacity. The constants in the model are measured, so they need re-measuring when the agent, the models or the call profile change.
If you're doing the same thing, what we'd recommend:
- Measure whether conversations complete, not whether calls connect.
- Chart median and tail latency together, and find your pipeline's latency floor before you commit to a target.
- Check host CPU whenever you check container throttling.
- Size on turns per minute rather than calls.
Charts marked as measured come from our own load tests. They are drawn on a relative scale: capacity figures are left out, and the shapes, ratios and percentages are real. Diagrams illustrate an idea and are not measured data.