A Step Towards the Best Voice AI Agent of India

We have been trying to build the best voice AI agent in India, from scratch, from first principles. This is a writeup of the stuff we have been working on.

The thing about a voice agent is that it is really two systems pretending to be one. There is the half you hear, which is the agent on the call, the turn taking, the interruptions, basically whether talking to it feels like talking to a person or like fighting an IVR. And there is the half you never hear, which is everything around the call, whether a real person even picked up, whether the call ran somewhere clean, whether we can even afford to run all of it. Both halves are first principles problems. And if either one is broken the whole thing is useless, infact a perfect sounding agent that nobody picks up is worth nothing, and a perfect call to a perfect agent that drops because some other call crashed the machine is also worth nothing.

Turn detection

Turn detection is the most important layer in the whole thing, and it is the one people get wrong the most. It is just the job of figuring out when the user has stopped talking and it is the agent’s turn to speak. Sounds trivial, it is not. If the detector is too patient the agent sits there after you finished and the whole thing feels slow and dead. If it is too eager it keeps cutting you off in the middle of your sentence and now every turn is a small fight. The entire game is finding the one that adds the least friction.

So instead of guessing I made a small eval set. I took real production calls plus a few competitor demos, and sat and annotated every user turn and agent turn by hand, across English and a bunch of Indian languages, around 800 turns. Then I ran a whole bunch of turn detectors on the exact audio we get off the phone line, which is low quality 8kHz mono, because it does not matter how a detector does on clean studio audio, it only matters how it does on what actually comes through a phone.

The thing you see immediately is that nobody is good at both.

The ones that almost never interrupt you also miss a lot of turns and leave dead air. The ones that catch every turn also fire on every cough and fan and background voice. There is no free lunch here, you are just picking a point on a curve. We shipped the one that caught turns well without being trigger happy, the red dot up in the top left.

One more thing worth knowing. There are basically two ways to detect a turn. One is acoustic, you just look at the sound, the pitch and the energy and the shape of the speech, and decide if the person went quiet. The other is semantic, you actually look at what the person said and decide if the sentence sounds finished. I think semantic is the right direction long term, because a pause is not just silence. Someone thinking in the middle of a sentence and someone who is actually done sound almost identical, the only difference is the meaning, and a pure acoustic detector has no way to tell them apart. It can hear that you stopped, it cannot hear that you were not done.

Interruption handling

Interruption handling is letting the user cut the agent off mid sentence and have it actually stop and listen. This sounds simple and it is genuinely hard, because it means every single stage of the pipeline has to be cancellable instantly, including the audio that is already coming out of the speaker. The model might still be generating, the voice might be mid word, there is audio buffered ahead. All of it has to be torn down the moment a real interruption lands. This is pure engineering, not a model problem.

But the harder half is not stopping the agent, it is deciding whether to stop it at all, because half the time the detector fires and the user did not actually mean to interrupt. These false interruptions are what make an agent feel twitchy and annoying to talk to. They come in two flavours.

1. Background noise getting heard as speech. A keyboard, a cough, a fan, a baby crying. The detector hears the energy and calls it a turn when nobody said anything.

2. Actual speech that is not the user. This one splits again. Sometimes it is the agent’s own voice leaking out of the speaker back into the mic, so the agent interrupts itself. Sometimes it is a second person in the room, and since there is no separation between the user’s voice and everyone else’s, a loud enough bystander triggers a turn. The real fix here is speaker separation, so the agent only ever listens to the one voice it is supposed to.

And then there is one edge case I find genuinely interesting. When the user interrupts, they did not hear the whole thing the agent was saying. The agent maybe generated three sentences but only two and a half actually made it out of the speaker before the cut. So if you build the next turn off the full text the agent generated, you are lying to yourself, you are assuming the user heard things they never heard. The fix is to use the word level timestamps from the speech to text to figure out exactly how far the audio actually got, work out what the user really heard, and build the next reply off that. Basically the agent’s memory of the conversation has to match the user’s memory of it. If it does not, the agent ends up referencing something the user never heard, and to the user the whole thing just feels broken, even when every individual piece is technically working.

Everything till here is about the agent once a call is already happening. But none of it matters until the call actually happens, and getting the call to happen, cleanly and cheaply and at scale, is the entire other half. This is the part nobody puts in a demo.

Getting the call connected

An agent that sounds perfect is useless if it never reaches anyone. We make a lot of outbound calls, and telephony does not work how you would assume. It works on channels. A channel is one live call you are allowed to have at a time, think of it like billing counters at a shop, it does not matter how long the line outside is, only so many people can be at a counter at once, and when they are all full everyone else just waits. We get a fixed number of these. And on top of that the carrier on the other side rejects most of our calls before they even ring properly.

So the dialing problem is not loop over a list and call dial. The real problem is, given a fixed pipe and a backlog roughly twice the size of the pipe, decide who to call, when, and how fast.

I pulled a normal day of production calls just to see where they actually go.

The large majority come straight back busy. A small chunk ring out with no answer. Only a small fraction actually connect to a human. And we are running the channels almost completely full the whole day. So the bottleneck is extremely clear, it is the channels and the carrier, it is not the dialer being clever.

This matters because I went into this to add predictive dialing, and predictive dialing turned out to solve a problem we do not have. Predictive dialing is a call center thing. Human agents are expensive, dialing has latency, most calls do not connect, so you dial ahead of your free agents so that by the time an agent finishes a call a new one has just connected, the whole point being to never let an expensive human sit idle. But our agent is not a human, it is software, and we can just make more of it when we need to. There is no idle human to keep busy. The thing predictive dialing optimises does not even exist for us. To chase our connect rate with classic over dialing you would need something like ten times the channels we have, and we cannot over dial past a wall.

I still built the machinery the right way, because the correct idea is to separate the two scarce things that were tangled together, the channels which are the pipe, and the live conversations which are the actual slots, grab a slot only when a call really answers, and never dial past the live channel headroom so it just falls back to plain first come first served when there is no room. So it can never make things worse, and it is ready for the day channels stop being the wall. But honestly, in our current situation it just behaves like first come first served, because we are channel bound.

Which means the things that actually moved the needle were two very boring things, not the fancy pacing.

1. We were capping our own channels by accident. There was a static limit sitting in the code that was lower than what we were actually paying the provider for, and a min() that meant the real higher limit could only ever pull the cap down, never up. So we were paying for a big pipe and using a fraction of it, a chunk of the channels were just sitting idle the whole time because of one number in a config. The fix was a one line change.

2. The busy rate. The large majority of calls coming back busy is the single biggest lever in the whole system and it has nothing to do with our code. The rejection happens on the carrier side, fairly evenly across all our numbers, which points at number reputation, not one bad number. Cutting that in half does more than any dialer change we could ever ship.

I think this is the most useful thing thinking from first principles buys you. The fancy named technique was sitting right there and it would have looked great on a slide, but the system was loudly telling me the bottleneck was a config typo and a number reputation problem. You only see that if you actually go and measure the thing in front of you instead of reaching for the textbook answer.

Two smaller things came out of the same reality. The dialer was pulling the oldest waiting calls globally, so one big early campaign would drain its whole backlog before a later campaign got a single line, which is real starvation when demand is double the capacity. So now it is round robin across campaigns, reshuffled every tick, and first come first served within a campaign. And when a call comes back busy we retry it, but not on a timer, because a timer just dumps more load on the same full channels at the same moment. Instead a busy call goes to the back of the queue and gets retried only when the backlog ahead of it drains. The queue position is the rate limiter, you do not need a separate clock fighting it.

Resilient Resource Allocation for each Call

Right now every live call runs as a handler inside a shared pool of workers on a few big shared machines, all the calls sharing the same processes on the same boxes. This is fine right up until one call misbehaves, a memory leak, a crash, a runaway loop, a bad tenant specific flow, anything, and because they are all in the same pool on the same machine it can take down every other call on that box with it. The blast radius of one bad call is the whole machine and everyone currently talking to it.

For something that is going to run thousands of calls at once across a bunch of different clients, each with their own flow and their own quirks, that is just the wrong shape. You want the opposite. You want every single call to get its own clean throwaway box. One call, one sandbox, and when the call ends you throw the sandbox away. If a call breaks something it breaks its own little box and nobody else even notices.

The obvious way to do this is to start a fresh box for every call. And the moment you try it you hit the wall that makes the whole idea hard.

Our voice process is heavy, it loads a pile of machine learning stuff on startup, and a cold boot takes long enough that the caller would hang up in dead air before the agent even says hello. So a fresh box per call is dead on arrival.

The way out is to not build it fresh each time. You boot one box once into a fully ready state, everything loaded and warm, take a snapshot of that whole state, and then start every per call box from the snapshot instead of from scratch. Restoring from a snapshot is roughly ten times faster, and more importantly it does not depend on the slow startup at all, because the startup already happened once when you took the snapshot. And because the heavy stuff is read only and identical across every call, the boxes can share those memory pages instead of each one paying for its own copy, so you get a real isolated box per call without paying full memory for each call. That is the difference between one box per call being a cute idea and it being something you can actually run at scale.

Then the part that actually mattered, the proof. It is easy to argue this works on a whiteboard. Voice is unforgiving though, so I drove real phone calls through it end to end and measured the audio against our normal setup.

The isolation added no real latency or jitter to the audio, the opening was if anything a touch faster, and there were no dropped frames. So the answer to can you put every single call in its own isolated box without hurting the audio is yes, I have run real calls through it and listened to them.

Having proven it out, this is now getting built properly into our own voice infra instead of sitting as a side experiment. And the cost moves the right way too, because instead of a fixed floor of big always on machines you end up paying much closer to per call.

Only paying for the day

Our voice service runs on a managed container platform, and the floor was a handful of big boxes running all day, every day. But here is the thing, we only make outbound calls during the day, in a fixed window. At night basically nothing happens. So overnight we had these big boxes sitting there fully provisioned and almost completely idle, and we paid for every one of those hours.

The fix in principle is obvious, only run full capacity during the window. So I put it on a schedule, scale up before the window opens so the boxes are warm in time, scale back down to a skeleton after it closes, and let the normal load based autoscaling handle spikes during the day. That part is easy.

The interesting part, the part that makes this a real systems problem and not a cron job, is scaling down. A web server is stateless, you can kill it whenever and the load balancer just sends the next request elsewhere. A call serving box is not like that. It is holding live calls, each one a person in the middle of a conversation. And the scheduler, when it decides to kill a box to scale down, has no idea which box has live calls on it. It just picks one and kills it.

This was actually already hurting us on every deploy, separate from the cost thing. The grace period a box gets to finish its existing connections before it gets killed was set way too short, much shorter than how long a call can run. So on every deploy and every scale down we were just guillotining any call that happened to be longer than that grace period. We were dropping live conversations as routine and it was hiding inside the word deploy.

The fix is to make that drain window longer than the longest call we ever have. Now when a box is marked to go, the load balancer immediately stops sending it new calls but lets the calls already on it run to the end, and since the window is longer than any call, they all finish naturally and then the box goes away. Nobody gets cut off. The scale down still happens, it just waits for the box to empty first.

There is a fancier version of this where the app itself tells the scheduler do not kill me I have live calls, and I actually built it, refcount and all. Then I threw it away, because the longer drain window already does the job and is much simpler. It only becomes necessary if our calls ever get longer than the drain window, so I left a note for that future and shipped the simple thing.

That is the second time in this whole project the boring version beat the clever version. Predictive dialing lost to a one line config fix, and the fancy scale down protection lost to a longer timeout. I do not think that is a coincidence. When you actually go and look, the real constraint is almost always simpler and dumber than the technique you were about to reach for.

Conclusion

None of the second half is the model, and that is the whole point I am trying to make. The first half, the turn detection and the interruptions, is the part you hear, the part that makes it feel like a person. The second half, the dialing and the isolation and the scheduling, is the part you never hear and never see in a demo. Whether the phone even connected, which turned out to be a carrier problem way more than a dialer problem. Whether the call ran somewhere clean. And whether we can actually afford to keep it on.

I think this half gets ignored because it does not demo well. You can show someone a snappy interruption in a ten second clip and they get it instantly, you cannot show them a busy carrier or a shared memory page or a drain window the same way. But this is the half that decides whether the agent actually works for a real person on a real phone at real scale, instead of just in the room where you are showing it off. The best sounding agent in the country is worth nothing if most of the calls come back busy, or if one bad call takes down ten good ones, or if the bill becomes the reason it gets shut down.

So that is the agent, both halves of it, getting it to sound like a person and getting it onto a real person’s phone cleanly and at a price we can pay. And honestly the systems half has been the most fun part, because the systems push back and tell you the truth, you just have to measure them and actually listen.