The problem
Voice agents shift state quickly. Voice alone is rarely enough for a strong user experience. On an outbound call - say placing an order or picking products - the agent must show matching visual cues and UI elements in real time. Shipping with confidence also means constant testing across customer profiles, accents, and intents.
We needed a two-fold solution:
- A voice-first agent that syncs verbal state with a visible UI.
- A unified evaluation platform that runs multi-turn chat and voice simulations and scores output without manual review.
Real-Time Voice-UI Synchronization
To link voice turns with on-screen app state, we built a custom orchestrator.
- WebSockets Sync: The orchestrator turns voice state changes - cart updates, checkout steps - into WebSocket payloads for the client.
- Live UI Rendering: The client renders cards, product grids, or forms from the agent's active LangGraph node.
- Interruption Handling: If the user cuts the agent off mid-sentence, the UI rolls back to the interrupted context.
The Agent Evaluation Platform
To test these agents at scale, we built a shared simulation and evaluation runner.
- Persona Simulation: A mock user agent chats or speaks with the bot using profiles like an angry shopper, a non-English speaker, or a failed-card checkout.
- LLM-as-a-Judge: After each run, a judge LLM scores the transcript on fluency, intent accuracy, relevance, and safety.
- Anomaly Detection & Tracing: The platform watches live metrics for phrase loops, drop-offs, latency spikes, and broken tool calls.
Platform-Agnostic Design
We decoupled agent logic from provider models with a shared configuration layer.
- Steps Configurator: Admins define nodes, transitions, and system instructions in a visual editor.
- Prompt Versioning: We track prompts in Git for rollbacks and side-by-side version tests.