Challenge
The first version of the phone line ran on an off-the-shelf telephony operator service with a language model and speech synthesis. After each utterance the caller waited about 1.5-2 s for a reply, which is a very long time in a phone conversation.
The diagnosis showed that the problem lay in how Polish was handled. The off-the-shelf service allowed only one speech recognition engine for Polish and detected the end of an utterance using a fixed 1.2 s window of silence. Newer models that allow faster replies did not support Polish.
Approach
First we broke the latency budget down stage by stage: end-of-utterance detection, transcription, the model's first word and speech synthesis. The first fixes covered the model configuration (reasoning switched off, short answers) and a timeout with one retry, so that a failure would not end in silence on the line.
We then compared four architectures: refining the existing one, an off-the-shelf voice agent platform, our own pipeline and external voice AI services. We chose our own pipeline because it gives full control over every stage and the lowest possible response time. The price is responsibility for the whole audio path.
Implementation
The new path: the audio stream from telephony goes to real-time speech recognition, then to voice activity detection and a semantic end-of-utterance model that supports Polish and runs on an ordinary CPU. Claude then generates the reply, and a speech synthesiser turns it into voice. The assistant recognises the caller by phone number and greets them personally, and every call carries a verified signature.
Test calls led to concrete fixes: the assistant no longer interrupts itself, it takes a phone number through a callback instead of by dictation, and it always confirms amounts, because speech recognition can confuse “siedem” (seven) with “siedemdziesiąt” (seventy).
Results
The new version has run in production as a pilot since July 2026. The change can be rolled back with a single commit. In test calls the speech recognition wrote digits correctly, and the remaining problems concerned the flow of the conversation, not the technology.
The next step is to measure latency on a series of identical calls before and after the change, so that the versions are compared on numbers and not on impressions.
