The problem
Voice agents shift state quickly. Voice alone is rarely enough for a strong user experience. On an outbound call—say placing an order or picking products—the agent must show matching visual cues and UI elements in real time. Shipping with confidence also means constant testing across customer profiles, accents, and intents.
We needed a two-fold solution:
- A voice-first agent that syncs verbal state with a visible UI.
- A unified evaluation platform that runs multi-turn chat and voice simulations and scores output without manual review.
Real-Time Voice-UI Synchronization
To link voice turns with on-screen app state, we built a custom orchestrator.
- WebSockets Sync: The orchestrator turns voice state changes—cart updates, checkout steps—into WebSocket payloads for the client.
- Live UI Rendering: The client renders cards, product grids, or forms from the agent's active LangGraph node.
- Interruption Handling: If the user cuts the agent off mid-sentence, the UI rolls back to the interrupted context.
The Agent Evaluation Platform
To test these agents at scale, we built a shared simulation and evaluation runner.
- Persona Simulation: A mock user agent chats or speaks with the bot using profiles like an angry shopper, a non-English speaker, or a failed-card checkout.
- LLM-as-a-Judge: After each run, a judge LLM scores the transcript on fluency, intent accuracy, relevance, and safety.
- Anomaly Detection & Tracing: The platform watches live metrics for phrase loops, drop-offs, latency spikes, and broken tool calls.
Platform-Agnostic Design
We decoupled agent logic from provider models with a shared configuration layer.
- Steps Configurator: Admins define nodes, transitions, and system instructions in a visual editor.
- Prompt Versioning: We track prompts in Git for rollbacks and side-by-side version tests.