Vaakyo x Cartesia: how our agents hear and speak

A voice agent has two jobs that people notice before anything else: it has to hear the caller properly, and it has to answer quickly in a voice that sounds like a person. Everything else, the prompt, the tools, the knowledge base, comes after.

That's why every new agent on Vaakyo starts with Cartesia on both ends: Ink listens and Sonic speaks. You can switch either one to Deepgram, Sarvam or ElevenLabs in the agent editor, and many teams do for particular languages. But Cartesia is the default, and this post explains why and how the pieces fit.

Listening: Ink knows when you've finished

The hardest part of listening on a phone call isn't the words. It's knowing when the caller has finished speaking. Reply too early and the agent talks over someone who was only pausing to think. Reply too late and the line goes quiet for so long that the caller says "hello?".

Most speech-to-text services give you words and leave the timing to you: you wait for a stretch of silence and guess. Cartesia's Ink streaming API works in turns instead. It sends:

• a turn.start when the caller begins;
• turn.update as the words come in;
• a turn.end when it judges the caller is done.

Ink detects the voice activity and the end of the turn itself, from how the speech sounds as well as the silence.

On Vaakyo that's the Turn detection setting on the Transcriber tab, with three presets:

• Responsive ends a turn soonest, for quick back-and-forth such as confirming an order.
• Balanced is the default.
• Patient waits longer, for callers who think aloud or read out long numbers.

Each preset is a set of Ink's thresholds and a timeout, tuned on real calls, so you choose a behaviour rather than a number.

Two more things matter on Indian calls:

• Language detection. Ink works out the language as the caller speaks, so a caller who starts in English and moves into Hindi doesn't confuse it.
• Key terms. Add up to 100 words the agent must catch, like product names, place names or your brand, and Ink listens for them.

Interruptions: the updates do the work

Because Ink sends what it has heard while the caller is still speaking, the agent doesn't have to wait for the end of a turn to react:

• The caller starts talking over the agent: the reply pauses.
• They only said "haan" or coughed: it carries on.
• They said a stop phrase, like "ruko", "wait" or "ek minute": the agent stops at once.

We wrote about this in how we detect interruptions. The short version is that it only works because the updates arrive early and often.

Speaking: Sonic streams as the model writes

The other half of a natural call is how fast the reply starts. On Vaakyo, the language model's answer goes to Sonic while it is still being written. Each call holds one connection to Sonic, and each reply is its own context on that connection. Text goes in sentence by sentence with continue: true, and audio starts coming back before the model has finished thinking. The caller hears the first words of a reply while the last ones are still being decided.

Contexts also make interruptions clean. When the caller cuts in:

• we cancel that reply's context, so Sonic stops making it;
• we drop any of its audio that has already arrived, by context id.

The next reply starts fresh on the same connection, with no reconnect.

Hindi, Hinglish and ten more languages

Sonic speaks twelve languages our customers' callers use: Hindi, English, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu and Urdu. Those are the languages every Vaakyo agent can pick from by default.

Hinglish needed one more step. Agents write Hinglish two ways: in Roman script ("aapka order kal aa jayega") or in Devanagari with English words mixed in. The two need different text normalisation, or numbers, dates and English words come out wrong. So for a Hindi voice, every piece of text sent to Sonic is checked:

• Devanagari is read with Hindi (hi-IN) rules;
• Roman script is read with Indian English (en-IN) rules.

The same agent can switch between the two mid-sentence and still sound like one person.

The voice's speed (0.6 to 1.5) is passed straight through. That is useful for agents that read out long details, or callers who want things slower.

Built for calls that can't drop

A voice service is a dependency in the middle of a live phone call, so we treat it like one:

• Backup voices. Each agent can list up to three. If its voice fails to connect, or fails mid-call, the next one takes over for the rest of the call, from any provider.
• Concurrency limits. Speech services cap how many streams can run at once, so Vaakyo counts every open stream across the platform. When one is full, the call moves to a backup that has room, and queued outbound calls wait for a free slot, instead of being refused mid-call. The details are in how we count speech slots with Redis leases.
• Usage on every call. Each call records how many characters it spoke and how many seconds it listened, so its cost breakdown shows exactly what it used.

Try it

Create an agent and press Talk: you'll be speaking to Ink and Sonic in the browser in under a minute. Then try the other transcribers and voices on the same agent, and use tests to compare them on your own scenarios. The best way to choose speech models is to hear them on your calls.