Sub-10ms Voice Agents in TypeScript: JEV, Zero Hallucinations with Felona Voice

The promise of voice AI is compelling: natural, intuitive interactions that streamline tasks and enhance user experience. Yet, the reality often falls short, plagued by frustrating latency, unexpected hallucinations, and rigid conversational flows. Traditional approaches, relying heavily on large language models (LLMs) for every turn, introduce significant delays (500ms-1200ms+) and an inherent unpredictability that undermines trust.

What if you could build voice agents that respond in sub-10 milliseconds, with absolute determinism and zero hallucinations? Enter Felona Voice, an innovative open-source TypeScript framework designed to revolutionize how we build ultra-low-latency voice agents.

The Latency and Hallucination Dilemma in Voice AI

For years, voice AI development has navigated a challenging landscape. On one hand, LLMs offer unparalleled flexibility and understanding of natural language. However, this comes at a steep price: the time it takes for an LLM to process input and generate a response is simply too high for truly fluid voice interactions. Those half-second to second-long pauses might seem minor, but they disrupt the natural rhythm of human conversation, leading to awkward silences and a perceived lack of intelligence.

Compounding this is the issue of LLM hallucinations. While powerful, LLMs can sometimes generate factually incorrect or nonsensical responses, especially when operating outside their training data or under ambiguous prompts. In a voice agent context, this can lead to frustrating miscommunications and a broken user experience.

Alternatively, developers might resort to hardcoded static conversational graphs. While these offer speed and determinism, they are inherently inflexible. Any deviation from the script, any unexpected user input, and the agent breaks down, unable to adapt to natural human speech patterns. This rigidity is a non-starter for truly intelligent conversational agents.

Felona Voice's Breakthrough: Joint Embedding Vectors (JEV) and VoiceGraph

Felona Voice tackles these fundamental challenges head-on with a groundbreaking approach powered by Joint Embedding Vectors (JEV) and stateful conversational transition graphs, dubbed VoiceGraph.

How JEV Powers Sub-10ms Decisions

Instead of relying on token-by-token LLM inference for every decision, Felona Voice leverages JEV similarity matching. Here's the magic:

1. Semantic Representation: Both your defined agent actions (e.g., "book a table," "check status") and the user's spoken input are converted into dense numerical representations called Joint Embedding Vectors. These vectors capture the semantic meaning of the text.
2. Instantaneous Matching: When a user speaks, their input's JEV is compared against the JEVs of your predefined actions. This similarity matching is an incredibly fast, computationally inexpensive operation, allowing Felona Voice to decide the next action in a remarkable ~5ms – often less than a tenth of the time an LLM would take!
3. Zero Token Latency, Zero Hallucinations: Because decisions are based on semantic similarity of pre-computed embeddings rather than generative text, there's no token generation latency. Furthermore, since the agent is deterministically routed to predefined actions, the risk of hallucinations is completely eliminated.

VoiceGraph: Stateful and Dynamic Flows

VoiceGraph orchestrates these JEV-powered decisions within a stateful conversational framework. Unlike rigid static graphs, VoiceGraph allows for dynamic transitions based on the semantic intent identified by JEVs. This means your agent can intelligently navigate complex conversations, understand context, and respond appropriately, all while maintaining sub-10ms decision times.

Let's put it into perspective:

Unparalleled Developer Experience

Felona Voice isn't just powerful under the hood; it's a joy to use for developers. The framework provides a fluent builder API that makes defining agents and their actions intuitive and readable.

Here’s how easy it is to get started:

import { createAgent } from "felona-voice";

const agent = createAgent("Concierge")
  .system("You are an intelligent voice concierge for a luxury hotel.")
  .action("book_table", "Book a restaurant reservation", async (ctx) => {
    // In a real app, you'd integrate with a booking system
    console.log(`User wants to book a table. Context: ${JSON.stringify(ctx.args)}`);
    return "Certainly, I can book a table for you. For how many people and at what time?";
  })
  .action("check_in_status", "Check guest check-in status", async (ctx) => {
    console.log(`User wants to check check-in status. Context: ${JSON.stringify(ctx.args)}`);
    return "Please provide your reservation number to check your check-in status.";
  })
  .fallback("I'm sorry, I didn't quite catch that. How can I assist you?");

// Simulate user interaction
async function runInteraction() {
  let reply = await agent.interact("Can I book a table?");
  console.log(`Agent: ${reply}`); // Expected: "Certainly, I can book a table for you. For how many people and at what time?"

  reply = await agent.interact("What's my check-in status?");
  console.log(`Agent: ${reply}`); // Expected: "Please provide your reservation number to check your check-in status."

  reply = await agent.interact("Tell me a joke.");
  console.log(`Agent: ${reply}`); // Expected: "I'm sorry, I didn't quite catch that. How can I assist you?"
}

runInteraction();

This example showcases the core .system(), .action(), and .fallback() methods. You define the agent's persona, its capabilities, and what to do if an intent isn't matched. The ctx object in action handlers provides access to context and arguments, allowing for sophisticated integrations.

Pluggable Audio Pipelines & Local Testing

Felona Voice is designed with flexibility in mind. It offers pluggable audio pipelines, supporting popular services like Deepgram, Whisper, ElevenLabs, and Cartesia, as well as standard WebSockets and WebRTC for real-time audio streaming. Crucially, you can develop and test your agent locally without needing any external API keys for deterministic routing, making the development loop incredibly fast and cost-effective.

Seamless Human Interruption Handling

One of the most challenging aspects of voice AI is gracefully handling human interruptions. With Felona Voice's deterministic, sub-10ms decision-making, the agent can quickly re-evaluate the user's intent even in mid-sentence. This means your voice agent can respond instantly to new commands or changes in topic, creating a truly natural, interruption-aware conversational flow.

Transforming Voice User Experience

The impact of sub-10ms decision latency cannot be overstated. It's the difference between an awkward, stilted interaction and a natural, fluid conversation. Users will perceive your voice agent as more intelligent, responsive, and ultimately, more helpful. By eliminating hallucinations and providing deterministic routing, Felona Voice builds trust and efficiency, opening up new possibilities for critical applications where accuracy and speed are paramount.

Imagine customer service agents that respond instantly, smart home devices that truly understand you, or voice assistants that feel like talking to another human. Felona Voice makes this a reality, providing the underlying framework for the next generation of conversational AI.

Get Started with Felona Voice Today!

Ready to build the future of voice AI? Felona Voice empowers you to create ultra-low-latency, zero-hallucination voice agents with an unmatched developer experience.

🌟 Star the repository on GitHub: https://github.com/mohitjoer/felona_voice 📦 Install via npm: npm install felona-voice 📖 Explore full documentation: https://felona-voice.mohitjoe.tech/docs