# Agent Evaluation & Voice-UI Sync

URL: https://mepritam.dev/work/agent-evaluation-and-voice-ui-platform/

Case study: Designing a voice-first agent (Gemini Live) with real-time UI screen synchronization, prompt versioning, and a unified agent evaluation framework.

---

### The problem

Voice agents shift state quickly. Voice alone is rarely enough for a strong user experience. On an outbound call—say placing an order or picking products—the agent must show matching **visual cues and UI elements** in real time. Shipping with confidence also means constant testing across customer profiles, accents, and intents.

We needed a two-fold solution:
1. A **voice-first agent** that syncs verbal state with a visible UI.
2. A **unified evaluation platform** that runs multi-turn chat and voice simulations and scores output without manual review.

### Real-Time Voice-UI Synchronization

To link voice turns with on-screen app state, we built a custom orchestrator.

- **WebSockets Sync:** The orchestrator turns voice state changes—cart updates, checkout steps—into WebSocket payloads for the client.
- **Live UI Rendering:** The client renders cards, product grids, or forms from the agent's active **LangGraph** node.
- **Interruption Handling:** If the user cuts the agent off mid-sentence, the UI rolls back to the interrupted context.

### The Agent Evaluation Platform

To test these agents at scale, we built a shared simulation and evaluation runner.

- **Persona Simulation:** A mock user agent chats or speaks with the bot using profiles like an angry shopper, a non-English speaker, or a failed-card checkout.
- **LLM-as-a-Judge:** After each run, a judge LLM scores the transcript on fluency, intent accuracy, relevance, and safety.
- **Anomaly Detection & Tracing:** The platform watches live metrics for phrase loops, drop-offs, latency spikes, and broken tool calls.

### Platform-Agnostic Design

We decoupled agent logic from provider models with a shared configuration layer.

- **Steps Configurator:** Admins define nodes, transitions, and system instructions in a visual editor.
- **Prompt Versioning:** We track prompts in Git for rollbacks and side-by-side version tests.