Sub-10ms Voice Agents in TypeScript: JEVs, State Machines & Felona Voice

The dream of truly natural, real-time conversational AI has long been hampered by one critical bottleneck: latency. Traditional Large Language Model (LLM) prompts, while powerful, introduce decision latencies ranging from 500ms to well over 1200ms per conversational turn. This delay, coupled with the notorious risk of hallucinations, shatters the illusion of seamless human-computer interaction, leaving users frustrated and applications feeling sluggish.

Enter Felona Voice, an innovative open-source TypeScript framework designed to revolutionize the landscape of voice AI. Felona Voice is engineered from the ground up to deliver ultra-low-latency, zero-hallucination voice agents, achieving decision times in a staggering sub-10ms range, often around ~5ms.

The Latency Dilemma: Why Traditional Approaches Fall Short

Imagine ordering a coffee or booking a flight using a voice assistant. Every pause, every slight delay, pulls you out of the experience. The current paradigm often involves:

1. Speech-to-Text (STT): User speaks, audio is converted to text.
2. LLM Processing: Text is sent to an LLM (e.g., OpenAI, Anthropic) for intent recognition and response generation.
3. Text-to-Speech (TTS): LLM's text response is converted back to audio.

Steps 2 and 3 are where the significant delays occur. LLM inference, especially for complex prompts, is inherently slow. Furthermore, while powerful, LLMs can hallucinate – generating plausible but incorrect information, which is unacceptable for critical applications like customer service or transactional systems. Hardcoded state machines, on the other hand, offer determinism but are too rigid, breaking down the moment a user deviates slightly from the script.

Felona Voice: A New Paradigm with Joint Embedding Vectors (JEV)

Felona Voice tackles these challenges head-on by introducing a groundbreaking approach centered around Joint Embedding Vectors (JEV) and a stateful conversational transition graph called VoiceGraph.

Instead of relying on token-based LLM inference for every decision, Felona Voice pre-computes semantic embeddings for all possible user intents and agent actions. When a user speaks, their utterance is quickly converted into an embedding. This user embedding is then compared against the pre-computed JEVs within the VoiceGraph.

The Magic of JEVs:

• Sub-10ms Decisions: Similarity matching between embedding vectors is an extremely fast mathematical operation, typically completing in milliseconds (~5ms). This eliminates the token latency associated with LLMs entirely.
• Zero Hallucinations: Because decisions are based on deterministic semantic similarity against predefined actions, there's no room for the agent to invent information. The system either finds a match or gracefully falls back.
• Contextual Understanding: JEVs capture the semantic meaning of utterances, allowing the system to understand variations in phrasing and intent without explicit keyword matching.

VoiceGraph: Deterministic Yet Flexible Conversation Flow

VoiceGraph is the intelligent backbone of Felona Voice. It's a state machine where each node represents a possible agent state or action. The transitions between these states are powered by JEV similarity matching. This offers the best of both worlds:

• Deterministic Routing: You define the possible actions and their semantic intent, ensuring predictable behavior.
• Natural Language Flexibility: Unlike rigid if/else or regex-based state machines, VoiceGraph, guided by JEVs, can intelligently match user utterances to the correct action even if the phrasing isn't an exact match.
• Seamless Interruption Handling: The low latency allows for immediate processing of user input, enabling highly responsive interruption handling, making conversations feel truly human-like.

Unparalleled Developer Experience

Felona Voice is built with developers in mind, offering a fluent, intuitive builder API in TypeScript:

import { createAgent } from "felona-voice";

// 1. Define your agent with a system prompt and actions
const conciergeAgent = createAgent("Concierge")
  .system("You are an intelligent voice concierge for a luxury hotel. Your primary role is to assist guests with bookings, information, and general inquiries. Always be polite and helpful.")
  .action("book_table", "Book a restaurant reservation for a guest.", async (ctx) => {
    // In a real application, you'd integrate with a booking system
    console.log(`Attempting to book a table for: ${ctx.utterance}`);
    // You can access context, parameters, etc., here
    return "Certainly, I've noted your request to book a table. What time and for how many people?";
  })
  .action("check_in_status", "Check the check-in status of a guest.", async (ctx) => {
    console.log(`Checking check-in status for: ${ctx.utterance}`);
    return "I can help with that. Could you please provide the guest's name or reservation number?";
  })
  .action("provide_directions", "Provide directions to a hotel amenity or local landmark.", async (ctx) => {
    console.log(`Providing directions for: ${ctx.utterance}`);
    return "Of course, where would you like directions to?";
  })
  .fallback("I'm sorry, I didn't quite catch that. How can I assist you today?");

// 2. Interact with your agent
async function runAgentDemo() {
  console.log("\n--- Felona Voice Concierge Demo ---\n");

  let reply = await conciergeAgent.interact("Can I book a table for two tonight?");
  console.log(`User: Can I book a table for two tonight?`);
  console.log(`Agent: ${reply}`); // Expected: "Certainly, I've noted your request..."

  reply = await conciergeAgent.interact("What's the status of my room?");
  console.log(`User: What's the status of my room?`);
  console.log(`Agent: ${reply}`); // Expected: "I can help with that. Could you please provide..."

  reply = await conciergeAgent.interact("Where is the nearest gym?");
  console.log(`User: Where is the nearest gym?`);
  console.log(`Agent: ${reply}`); // Expected: "Of course, where would you like directions to?"

  reply = await conciergeAgent.interact("Tell me a joke.");
  console.log(`User: Tell me a joke.`);
  console.log(`Agent: ${reply}`); // Expected: "I'm sorry, I didn't quite catch that..." (fallback)
}

runAgentDemo();

This example demonstrates how straightforward it is to define complex conversational logic. Each .action() defines an intent (via its description) and the corresponding asynchronous function to execute. The .fallback() method ensures a graceful response for unhandled utterances.

Pluggable Audio Pipelines & Local Testing

Felona Voice isn't just about decision making; it's a complete framework. It supports a wide array of pluggable audio pipelines, including:

• WebSockets and WebRTC for real-time browser-based communication.
• Integrations with leading STT/TTS providers like Deepgram, Whisper, ElevenLabs, and Cartesia.

Crucially, Felona Voice allows for zero external API keys for local testing and deterministic routing. This means you can build, test, and iterate on your voice agents locally without incurring costs or relying on external services until deployment.

Why Sub-10ms Matters for Voice UX

The difference between 500ms and 5ms is profound. It transforms the user experience from a disjointed, robotic interaction into a fluid, human-like conversation. Users no longer have to wait awkwardly for the agent to process their request. This immediate feedback loop fosters trust, reduces cognitive load, and significantly enhances user satisfaction.

The Future of Voice AI is Here

Felona Voice pushes the boundaries of what's possible in voice AI. By leveraging the power of Joint Embedding Vectors and intelligent state machines, it delivers an unparalleled combination of speed, accuracy, and developer-friendliness.

Whether you're building sophisticated customer service bots, interactive voice assistants, or novel voice-controlled applications, Felona Voice provides the robust, high-performance foundation you need to create truly engaging and effective conversational experiences.

Ready to build voice agents that feel truly alive?

• 🌟 Star the repository on GitHub: https://github.com/mohitjoer/felona_voice
• 📦 Install via npm: npm install felona-voice
• 📖 Explore full documentation: https://felona-voice.mohitjoe.tech/docs