Sub-10ms Voice Agents: TypeScript, JEVs & Zero-Hallucination State Machines

Voice agents are no longer a novelty; they're a critical interface for myriad applications. However, the promise of truly natural, real-time voice interaction has often been hampered by two persistent challenges: latency and unpredictable responses, commonly known as 'hallucinations' in the age of Large Language Models (LLMs). Achieving a sub-10ms response time for a voice agent, while simultaneously eliminating generative AI's propensity to invent answers, requires a fundamentally different architectural approach.

This deep dive explores how to build ultra-low-latency, zero-hallucination voice agents in TypeScript. We'll combine the power of Joint Embedding Vectors (JEVs) for lightning-fast, deterministic action prediction, robust state machines for conversation flow, and WebSockets for real-time audio streaming, all while ensuring seamless human interruption handling.

The Latency and Hallucination Conundrum

Traditional LLM-based voice agents face inherent limitations in real-time scenarios. The pipeline typically involves:

1. Speech-to-Text (STT): User speaks, audio is transcribed.
2. LLM Processing: Text is sent to an LLM for intent recognition, context understanding, and response generation.
3. Text-to-Speech (TTS): LLM's text response is synthesized into audio.

Each step introduces latency. STT takes time to process audio chunks, LLM inference can range from hundreds of milliseconds to several seconds depending on model size and complexity, and TTS adds further delay. Compounding this, LLMs, by design, are probabilistic generative models. While powerful, this means they can 'hallucinate'—generate plausible but incorrect or irrelevant information—which is unacceptable for critical applications like customer service, control systems, or medical assistance.

Our goal is to circumvent these issues by predicting the next action or next state directly from user input in milliseconds, without engaging a generative LLM for every turn.

Joint Embedding Vectors (JEVs): Sub-5ms, Zero-Hallucination Prediction

The core of our sub-10ms strategy lies in Joint Embedding Vectors (JEVs). Unlike sending raw text to an LLM, JEVs operate in a highly optimized, pre-trained vector space. The concept is simple yet powerful:

• Joint Representation: User utterances, agent capabilities (actions), and contextual states are all represented as vectors in a shared, high-dimensional embedding space.
• Similarity Search: When a user speaks, their utterance's embedding is generated. The agent then performs a rapid similarity search within its pre-defined action/state embedding space to find the closest match.
• Deterministic Prediction: This similarity search is a classification problem, not a generation problem. The closest matching vector deterministically points to a pre-programmed next action or state. This inherently eliminates LLM hallucinations because the agent is not generating new text; it's selecting from a finite, pre-defined set of responses or actions.

How it Achieves ~5ms Prediction

1. Optimized Embedding Models: The model that generates the user utterance embedding is typically a smaller, highly optimized transformer or recurrent neural network (RNN) designed for low-latency inference on edge devices or specialized hardware. This can run in milliseconds.
2. Vector Databases/Indices: The pre-computed embeddings for all possible agent actions and states are stored in an efficient vector database (e.g., Faiss, HNSWlib, Pinecone) or an in-memory index. Similarity search (e.g., cosine similarity) on these indices is incredibly fast, often completing in sub-5ms for thousands or even millions of vectors.
3. No Generative Inference: The critical factor is avoiding the computationally expensive token-by-token generation process of an LLM.

Practical JEV Implementation (Conceptual)

Training a JEV model is beyond the scope of this article, but conceptually, you'd train a model to embed pairs of (user_utterance, context, desired_action) into a shared space such that similar pairs are close together. For our runtime, we assume we have a pre-trained embeddingService and a vectorIndex.

// Assuming a pre-trained embedding service and a vector index
interface EmbeddingService {
  embed(text: string): Promise<number[]>;
}

interface VectorIndex {
  search(queryVector: number[], k: number): Promise<{ id: string; score: number }[]>;
}

// Mock implementation for demonstration
class MockEmbeddingService implements EmbeddingService {
  async embed(text: string): Promise<number[]> {
    // Simulate embedding generation time
    await new Promise(resolve => setTimeout(resolve, 2)); 
    // In a real scenario, this would call a model endpoint or run an on-device model
    const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0);
    return [hash % 100 / 100, (hash * 7) % 100 / 100, (hash * 13) % 100 / 100]; // Dummy embedding
  }
}

class MockVectorIndex implements VectorIndex {
  private actionEmbeddings: Map<string, number[]> = new Map();

  constructor() {
    // Pre-populate with some mock actions and their embeddings
    this.actionEmbeddings.set('greet_user', [0.5, 0.2, 0.8]);
    this.actionEmbeddings.set('ask_for_name', [0.1, 0.7, 0.3]);
    this.actionEmbeddings.set('confirm_order', [0.9, 0.1, 0.4]);
    this.actionEmbeddings.set('cancel_order', [0.8, 0.1, 0.5]);
    this.actionEmbeddings.set('provide_help', [0.3, 0.6, 0.1]);
  }

  async search(queryVector: number[], k: number = 1): Promise<{ id: string; score: number }[]> {
    // Simulate vector search time
    await new Promise(resolve => setTimeout(resolve, 3)); 
    
    let bestMatch: { id: string; score: number } | null = null;
    let maxScore = -1;

    for (const [id, embedding] of this.actionEmbeddings.entries()) {
      const score = this.cosineSimilarity(queryVector, embedding);
      if (score > maxScore) {
        maxScore = score;
        bestMatch = { id, score };
      }
    }
    return bestMatch ? [bestMatch] : [];
  }

  private cosineSimilarity(vec1: number[], vec2: number[]): number {
    const dotProduct = vec1.reduce((sum, val, i) => sum + val * vec2[i], 0);
    const magnitude1 = Math.sqrt(vec1.reduce((sum, val) => sum + val * val, 0));
    const magnitude2 = Math.sqrt(vec2.reduce((sum, val) => sum + val * val, 0));
    if (magnitude1 === 0 || magnitude2 === 0) return 0;
    return dotProduct / (magnitude1 * magnitude2);
  }
}

// Usage example:
async function predictNextAction(utterance: string): Promise<string | null> {
  const embeddingService = new MockEmbeddingService();
  const vectorIndex = new MockVectorIndex();

  const utteranceEmbedding = await embeddingService.embed(utterance);
  const results = await vectorIndex.search(utteranceEmbedding, 1);

  if (results.length > 0 && results[0].score > 0.7) { // Threshold for confidence
    console.log(`Predicted action: ${results[0].id} with score: ${results[0].score}`);
    return results[0].id;
  } else {
    console.log('No confident action predicted.');
    return 'fallback_to_clarification';
  }
}

// Example calls (in a real system, this would be driven by STT output)
predictNextAction("Hi there").then(action => console.log('Action for "Hi there":', action));
predictNextAction("I want to cancel my order").then(action => console.log('Action for "I want to cancel my order":', action));
predictNextAction("Tell me a joke").then(action => console.log('Action for "Tell me a joke":', action));

Deterministic State Machines for Conversation Flow

While JEVs provide rapid action prediction, a voice agent needs a structured way to manage conversation flow, context, and state transitions. This is where a deterministic Finite State Machine (FSM) or Hierarchical State Machine (HSM) becomes indispensable. It ensures that the agent behaves predictably and can recover gracefully from unexpected input.

Why State Machines?

• Predictability: The agent's next state is always a direct consequence of its current state and the predicted action (event).
• Zero Hallucination (Flow): The state machine, by definition, only allows transitions to predefined states and execution of predefined actions. There's no room for the agent to 'invent' a new conversational path.
• Context Management: Each state can implicitly hold or manage specific context relevant to that part of the conversation.
• Error Handling: Invalid transitions or unhandled events can be explicitly caught and managed (e.g., transition to a clarify state).
• Maintainability: Complex conversational logic is broken down into manageable states and transitions.

TypeScript State Machine Implementation

We can define states and events using TypeScript enums and types, then implement a transition function.

enum AgentState {
  IDLE = 'IDLE',
  GREETING = 'GREETING',
  ASKING_NAME = 'ASKING_NAME',
  MAIN_MENU = 'MAIN_MENU',
  ORDER_FLOW = 'ORDER_FLOW',
  CONFIRM_ORDER = 'CONFIRM_ORDER',
  CANCEL_ORDER = 'CANCEL_ORDER',
  PROVIDING_HELP = 'PROVIDING_HELP',
  CLARIFYING = 'CLARIFYING',
  CLOSING = 'CLOSING',
}

enum AgentEvent {
  USER_GREETS = 'USER_GREETS',
  USER_PROVIDES_NAME = 'USER_PROVIDES_NAME',
  USER_ASKS_FOR_MENU = 'USER_ASKS_FOR_MENU',
  USER_WANTS_TO_ORDER = 'USER_WANTS_TO_ORDER',
  USER_CONFIRMS = 'USER_CONFIRMS',
  USER_CANCELS = 'USER_CANCELS',
  USER_ASKS_FOR_HELP = 'USER_ASKS_FOR_HELP',
  USER_INTERRUPTS = 'USER_INTERRUPTS',
  SYSTEM_TIMEOUT = 'SYSTEM_TIMEOUT',
  ACTION_COMPLETED = 'ACTION_COMPLETED',
  NO_CONFIDENT_MATCH = 'NO_CONFIDENT_MATCH',
}

interface AgentContext {
  userName?: string;
  currentOrder?: string;
  lastUserUtterance?: string;
  // ... other conversational data
}

interface StateTransition {
  from: AgentState;
  event: AgentEvent;
  to: AgentState;
  action?: (context: AgentContext) => Promise<void>;
}

const transitions: StateTransition[] = [
  { from: AgentState.IDLE, event: AgentEvent.USER_GREETS, to: AgentState.GREETING, action: async (ctx) => { console.log('Agent: Hello! What can I do for you?'); } },
  { from: AgentState.GREETING, event: AgentEvent.ACTION_COMPLETED, to: AgentState.MAIN_MENU },
  { from: AgentState.MAIN_MENU, event: AgentEvent.USER_WANTS_TO_ORDER, to: AgentState.ORDER_FLOW, action: async (ctx) => { console.log('Agent: Sure, what would you like to order?'); } },
  { from: AgentState.ORDER_FLOW, event: AgentEvent.USER_CONFIRMS, to: AgentState.CONFIRM_ORDER, action: async (ctx) => { console.log('Agent: Confirming your order...'); } },
  { from: AgentState.MAIN_MENU, event: AgentEvent.USER_CANCELS, to: AgentState.CANCEL_ORDER, action: async (ctx) => { console.log('Agent: What would you like to cancel?'); } },
  { from: AgentState.MAIN_MENU, event: AgentEvent.USER_ASKS_FOR_HELP, to: AgentState.PROVIDING_HELP, action: async (ctx) => { console.log('Agent: How can I assist you?'); } },
  { from: AgentState.MAIN_MENU, event: AgentEvent.NO_CONFIDENT_MATCH, to: AgentState.CLARIFYING, action: async (ctx) => { console.log('Agent: I didn\'t quite get that. Could you please rephrase?'); } },
  // ... add more transitions

  // Global interruption handling example
  { from: AgentState.GREETING, event: AgentEvent.USER_INTERRUPTS, to: AgentState.MAIN_MENU, action: async (ctx) => { console.log('Agent: Apologies, how can I help?'); } },
  { from: AgentState.ORDER_FLOW, event: AgentEvent.USER_INTERRUPTS, to: AgentState.MAIN_MENU, action: async (ctx) => { console.log('Agent: What else can I do for you?'); } },
];

class AgentStateMachine {
  private currentState: AgentState = AgentState.IDLE;
  private context: AgentContext = {};

  constructor(initialState: AgentState = AgentState.IDLE) {
    this.currentState = initialState;
  }

  public getCurrentState(): AgentState {
    return this.currentState;
  }

  public getContext(): AgentContext {
    return this.context;
  }

  public async transition(event: AgentEvent, userUtterance?: string): Promise<void> {
    const possibleTransitions = transitions.filter(
      (t) => t.from === this.currentState && t.event === event
    );

    if (possibleTransitions.length === 0) {
      console.warn(`No transition found for state ${this.currentState} with event ${event}.`);
      // Fallback: maybe transition to a clarification state or error state
      return;
    }

    const transition = possibleTransitions[0]; // Take the first matching transition
    this.currentState = transition.to;
    this.context.lastUserUtterance = userUtterance; // Update context

    console.log(`Transitioned from ${transition.from} to ${transition.to} via event ${event}`);

    if (transition.action) {
      await transition.action(this.context);
    }
  }
}

// Usage:
const agent = new AgentStateMachine();

async function simulateInteraction() {
  await agent.transition(AgentEvent.USER_GREETS, "Hello"); // JEV would predict USER_GREETS
  await agent.transition(AgentEvent.ACTION_COMPLETED); // Agent finishes greeting
  await agent.transition(AgentEvent.USER_WANTS_TO_ORDER, "I want to buy a coffee"); // JEV predicts USER_WANTS_TO_ORDER
  // ... more interactions
}

simulateInteraction();

WebSockets for Real-time Audio Streaming

For truly low-latency voice interaction, traditional request-response HTTP models are insufficient. WebSockets provide a persistent, full-duplex communication channel ideal for streaming audio in real-time.

Why WebSockets?

• Persistent Connection: No overhead of establishing new connections for each audio chunk.
• Full-Duplex: Both client and server can send data simultaneously, crucial for handling human interruption.
• Low Latency: Minimal protocol overhead compared to HTTP.

Audio Streaming Strategy

1. Client-side Audio Capture: Use MediaRecorder API (browser) or audio libraries (Node.js) to capture audio.
2. Chunking and Encoding: Break audio into small chunks (e.g., 20-50ms) and encode them (e.g., Opus, Speex, or even raw PCM for minimal overhead). Smaller chunks reduce latency but increase network overhead. Opus is a good balance.
3. WebSocket Transmission: Send encoded audio chunks over the WebSocket connection.
4. Server-side Processing: The server receives chunks, passes them to a low-latency STT (e.g., Vosk, DeepSpeech, or cloud STT with streaming API), aggregates recognized text, and feeds it to the JEV prediction engine.
5. Agent Response: Once an action is predicted and the state machine transitions, the agent generates a TTS response, chunks it, and streams it back to the client via the same WebSocket.

// Server-side WebSocket (using ws library)
import { WebSocket, WebSocketServer } from 'ws';
import { AgentStateMachine, AgentEvent, AgentState } from './stateMachine'; // Assuming stateMachine.ts
import { predictNextAction } from './jevService'; // Assuming jevService.ts

interface ClientState {
  sttBuffer: string[];
  lastSttResult: string;
  agent: AgentStateMachine;
  // ... other client-specific data
}

const wss = new WebSocketServer({ port: 8080 });

const clients = new Map<WebSocket, ClientState>();

wss.on('connection', ws => {
  console.log('Client connected');
  clients.set(ws, { 
    sttBuffer: [], 
    lastSttResult: '',
    agent: new AgentStateMachine() // Each client gets its own agent instance
  });

  ws.on('message', async message => {
    const clientState = clients.get(ws);
    if (!clientState) return;

    // In a real scenario, 'message' would be audio bytes
    // For this example, let's assume it's transcribed text from a streaming STT
    const transcribedText = message.toString(); 
    clientState.sttBuffer.push(transcribedText);

    // Simulate STT processing and aggregation
    // In reality, a streaming STT would give you partial and final results.
    // Here, we'll just check if the buffer has enough text to process.
    if (clientState.sttBuffer.join(' ').length > 5 && transcribedText.endsWith('.')) { // Simple heuristic
      const fullUtterance = clientState.sttBuffer.join(' ').trim();
      clientState.sttBuffer = []; // Clear buffer
      clientState.lastSttResult = fullUtterance;

      console.log(`Received utterance: "${fullUtterance}"`);

      // JEV predicts the event
      const predictedAction = await predictNextAction(fullUtterance);
      let agentEvent: AgentEvent;

      switch (predictedAction) {
        case 'greet_user': agentEvent = AgentEvent.USER_GREETS; break;
        case 'confirm_order': agentEvent = AgentEvent.USER_CONFIRMS; break;
        case 'cancel_order': agentEvent = AgentEvent.USER_CANCELS; break;
        case 'provide_help': agentEvent = AgentEvent.USER_ASKS_FOR_HELP; break;
        case 'fallback_to_clarification': agentEvent = AgentEvent.NO_CONFIDENT_MATCH; break;
        default: agentEvent = AgentEvent.NO_CONFIDENT_MATCH; break; // Default to clarification
      }

      // Transition state and execute action
      await clientState.agent.transition(agentEvent, fullUtterance);

      // Simulate TTS response based on current state (simplified)
      let agentResponse = "";
      switch (clientState.agent.getCurrentState()) {
        case AgentState.GREETING: agentResponse = "Hello! How can I assist you today?"; break;
        case AgentState.MAIN_MENU: agentResponse = "I can help with orders, cancellations, or provide general support."; break;
        case AgentState.CLARIFYING: agentResponse = "I didn't quite catch that. Could you please say it again?"; break;
        // ... more responses
        default: agentResponse = "Hmm, I'm not sure how to respond to that."; break;
      }

      // In a real system, this would be actual TTS audio chunks
      ws.send(JSON.stringify({ type: 'agent_response', text: agentResponse }));
    }
  });

  ws.on('close', () => {
    console.log('Client disconnected');
    clients.delete(ws);
  });

  ws.on('error', error => {
    console.error('WebSocket error:', error);
  });
});

console.log('WebSocket server started on port 8080');

Seamless Human Interruption Handling

A truly natural voice agent must allow users to interrupt at any time, just like in human conversation. This is crucial for maintaining a sub-10ms perceived latency and a fluid user experience.

How to Implement Interruption:

1. Voice Activity Detection (VAD): On the client side, continuously monitor incoming audio for voice activity. If the user starts speaking while the agent is speaking, detect this.
2. Server-side STT Prioritization: When VAD detects user speech, immediately signal the server. The server's streaming STT should prioritize processing the new user audio. Any ongoing agent TTS should be immediately stopped or faded out.
3. State Machine Interruption Event: The USER_INTERRUPTS event in our state machine becomes critical. When an interruption is detected, the JEV processes the partial (or full) user utterance, and the state machine transitions to an appropriate interrupted or main_menu state, allowing the user to take control.

Client-side VAD & Interruption Logic (Conceptual):

// Example: Client-side audio processing and interruption signal
// This is highly simplified and assumes a VAD library/service

class AudioStreamer {
  private ws: WebSocket;
  private mediaRecorder: MediaRecorder | null = null;
  private audioContext: AudioContext;
  private analyser: AnalyserNode;
  private stream: MediaStream | null = null;
  private isAgentSpeaking: boolean = false; // Flag to track agent's TTS output

  constructor(websocketUrl: string) {
    this.ws = new WebSocket(websocketUrl);
    this.ws.onmessage = this.handleAgentResponse.bind(this);
    this.audioContext = new (window.AudioContext || (window as any).webkitAudioContext)();
    this.analyser = this.audioContext.createAnalyser();
    this.analyser.fftSize = 256;
  }

  async startStreaming() {
    this.stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const source = this.audioContext.createMediaStreamSource(this.stream);
    source.connect(this.analyser);

    this.mediaRecorder = new MediaRecorder(this.stream, { mimeType: 'audio/webm; codecs=opus' });
    this.mediaRecorder.ondataavailable = (event) => {
      if (event.data.size > 0) {
        this.ws.send(event.data); // Send audio chunk to server
      }
    };
    this.mediaRecorder.start(50); // Send 50ms audio chunks

    // Periodically check for user voice activity
    setInterval(this.checkVoiceActivity.bind(this), 100);
  }

  stopStreaming() {
    this.mediaRecorder?.stop();
    this.stream?.getTracks().forEach(track => track.stop());
  }

  private checkVoiceActivity() {
    const dataArray = new Uint8Array(this.analyser.fftSize);
    this.analyser.getByteFrequencyData(dataArray);
    const sum = dataArray.reduce((a, b) => a + b, 0);
    const average = sum / dataArray.length;

    // Simple VAD: If average volume above a threshold AND agent is speaking, signal interruption
    const VAD_THRESHOLD = 20; // Adjust based on microphone sensitivity and environment
    if (average > VAD_THRESHOLD && this.isAgentSpeaking) {
      console.log('User interruption detected!');
      this.ws.send(JSON.stringify({ type: 'interruption_signal' }));
      this.isAgentSpeaking = false; // Assume user has taken over
      // Optionally, stop agent's TTS playback here
    }
  }

  private handleAgentResponse(event: MessageEvent) {
    const data = JSON.parse(event.data as string);
    if (data.type === 'agent_response') {
      console.log('Agent says:', data.text);
      this.isAgentSpeaking = true;
      // In a real app, you'd play this TTS audio and set isAgentSpeaking to false when finished.
      setTimeout(() => this.isAgentSpeaking = false, data.text.length * 50); // Simulate TTS duration
    }
    if (data.type === 'interruption_signal_ack') {
        // Server acknowledged interruption, agent stopped speaking
        this.isAgentSpeaking = false;
    }
  }
}

// To use:
// const streamer = new AudioStreamer('ws://localhost:8080');
// streamer.startStreaming();

On the server, upon receiving an interruption_signal, the WebSocket handler would trigger clientState.agent.transition(AgentEvent.USER_INTERRUPTS). The state machine can then decide how to handle it based on its current state.

Putting It All Together: Architecture Overview

This architecture creates a highly responsive loop:

1. Client: Captures user audio, performs VAD, streams chunks via WebSocket.
2. Server (STT): Receives audio chunks, processes with low-latency streaming STT, aggregates text.
3. Server (JEV): Takes aggregated text (and current conversation context), generates embedding, performs vector search, predicts AgentEvent (~5ms).
4. Server (State Machine): Receives AgentEvent, transitions state, executes associated action (e.g., database lookup, API call).
5. Server (TTS): Generates TTS audio for the agent's response, streams chunks back to client via WebSocket.
6. Client: Plays TTS audio. If VAD detects user speech during agent TTS, an interruption signal is sent, restarting the loop from step 1, potentially bypassing steps 2-4 if the interruption is critical.

This tight loop, with JEVs replacing generative LLMs for core decision-making, allows for sub-10ms response times for critical turns, where the agent needs to quickly acknowledge or react.

Practical Tips and Considerations

• JEV Model Training: This is the most complex part. You'll need a diverse dataset of user utterances, contexts, and desired agent actions/responses. Fine-tuning an existing embedding model (e.g., from Hugging Face) on your domain-specific data is a common approach.
• Thresholding: The results[0].score > 0.7 in our JEV example is crucial. A low confidence score should trigger a NO_CONFIDENT_MATCH event, leading to a CLARIFYING state rather than a hallucinated action.
• Contextual Embeddings: For richer interactions, the JEV should embed not just the user utterance, but also the current state and recent conversational history. This can be achieved by concatenating embeddings or using a more sophisticated contextual embedding model.
• Hybrid Approach: For highly complex or open-ended queries that fall outside the defined JEV actions, you can still fall back to a traditional LLM. The state machine would manage this transition, perhaps to a LLM_QUERY state, but this would incur higher latency and potential for hallucination.
• Edge vs. Cloud: Deploying the STT and JEV embedding model on the edge (e.g., WebAssembly in browser, or on-device for mobile) can further reduce latency by minimizing network round-trips.
• Scalability: WebSockets are efficient, but managing many concurrent connections requires a robust server infrastructure (Node.js clusters, load balancers).
• Error Handling: Implement comprehensive error handling for network issues, STT failures, and unexpected JEV predictions.
• Audio Codecs: Choose efficient, low-latency audio codecs like Opus for streaming.

Conclusion

Building sub-10ms, zero-hallucination voice agents in TypeScript is an ambitious but achievable goal. By strategically replacing generative LLMs with Joint Embedding Vectors for core decision-making, leveraging deterministic state machines for robust conversational flow, and employing WebSockets for real-time audio, we can create voice experiences that feel truly instantaneous and reliable. This architecture pushes the boundaries of real-time AI, paving the way for a new generation of highly responsive and trustworthy intelligent agents.