Speech vendors have concurrency limits: counting slots with Redis leases

Every live call holds at least two streaming connections to speech vendors: a transcriber listening to the caller, and a voice speaking the agent's replies. Vendors cap how many of those you may run at once, depending on your plan, and they don't queue you politely when you go over. They refuse: an error on connect, or a socket closed mid-call.

A refused transcriber means the agent goes deaf mid-call. A refused voice means silence. So we stopped finding out the limit from the error and started counting.

Leases in a sorted set

Each provider and kind (speech-to-text, text-to-speech) has a limit, set by the platform team from the vendor's plan. Usage is counted in Redis, shared by every API process, including both colours during a deploy.

A slot is a lease: a member of a Redis sorted set, scored by its expiry time. Taking one is a single Lua script that:

1. removes members whose expiry has passed;
2. counts what's left;
3. adds the new lease if the count is below the limit.

Because it's one script, two processes can't both take the last slot. Because leases expire (90 seconds for a transcriber, 60 for a voice), a process that crashes frees its slots on its own within a minute or so. Live processes refresh their own leases with a heartbeat every 20 seconds.

Transcribers: one slot per call

A transcriber stream lasts the whole call, so it takes one slot when the call connects and gives it back when the call ends.

If the agent's transcriber is at its limit and the agent has a fallback transcriber from another provider with room, the fallback becomes the primary for that call. If nothing has room, we connect to the primary anyway: an agent that might be refused beats an agent that certainly can't hear.

Voices: one slot per reply

Voice limits count simultaneous generations, not open connections. So each reply takes a slot when it starts speaking and releases it when all its audio has arrived, or when it's cut off. Many calls can share a vendor's limit, because callers spend much of a call talking, not listening.

When the agent's voice provider is full, that one reply moves to a fallback voice from another provider with room, which is connected on demand and kept for later overflows. If none has room, the reply waits up to a second and a half for a slot, then goes ahead on the full provider rather than leave the caller in silence.

Each time this happens, the call's timeline gets a warning event, so overflow shows up in the call it affected.

Don't dial what you can't serve

Inbound calls are answered immediately; a ringing phone can't wait. Outbound calls can. Before the dispatcher dials a queued call, it checks that one of the agent's voice providers has room, and reserves a transcriber slot under the call's id. When the call connects, the call takes that reservation over. If it's never answered, the reservation is dropped when the call ends.

If there's no room, the call stays queued with waiting_for: "provider capacity", visible in the console, and is dialled when a slot frees up. Every call that ends prompts the dispatcher to look again, and a background job retries every 10 seconds.

Fail open

Redis is a dependency of the capacity counter, not of a phone call. If Redis is unreachable, every lease is granted uncounted, a warning is logged once a minute, and calls go ahead. Capacity code never raises an error into a call. Over-booking a vendor for a few minutes is better than refusing every call.

Limits that learn, carefully

Limits come from the vendors' published plans. If a vendor refuses us for too many streams at a lower count than we expected, we lower that provider's limit to what was in use minus one, and alert the platform team. We never raise a limit automatically. Being too careful costs a few seconds in the queue; being too generous costs dropped calls.

We also record the peak usage per minute for each provider, so the team can see how close each one runs to its limit and upgrade a plan before calls start waiting.