The problem
Voice interfaces in production are fragile. Chat users tolerate small delays or formatting issues. Voice needs sub-second latency and strict safety. A wrong price, a bad tool call, or off-topic drift on a support call damages trust and revenue at once.
We needed stateful phone orchestration, safe backend tools (order lookup, delivery reschedule), and brand-safe replies—with full audit trails.
System Architecture
The voice bot has three decoupled layers.
- Orchestration & Voice Gateway: PyCAT handles SIP/telephony and audio streams, wired into a stateful LangGraph flow.
- Safety & Guardrail Engine: Low-latency classifiers catch prompt injection, PII leaks, toxicity, and topic drift before the LLM and before speech synthesis.
- Execution & Auditing Layer: A sandbox runs actions, logs each turn, and emits OpenTelemetry spans.
Guardrails and Safe Tool Execution
To run tools safely for the customer, we added a strict schema-validation layer with Zod and JSON Schema.
- Strict Validation: Every tool call is schema-checked before it hits backend services.
- Rate-Limiting & Retries: Safeguards shield internal APIs from cascading failures under load.
- Turn Guardrails: Evaluation nodes check user intent and bot replies. Off-topic turns get a polite fallback.
Observability: Tracing and Audit Logs
To debug latency and conversation flow, we added tracing and structured logs.
- OpenTelemetry Tracing: Spans cover STT, NLU, LLM generation, tool calls, and TTS.
- Structured Audit Logging: Each session writes a JSON timeline of states, variables, and tool params for fast failed-call debugging.