Unlock Sub-10ms Voice AI: Real-time Agents with TypeScript & Felona Voice

The dream of truly real-time, human-like voice agents has long been hampered by one critical bottleneck: latency. Traditional voice AI systems, heavily reliant on large language models (LLMs) for conversational understanding and response generation, introduce significant delays. We're talking 500ms to 1200ms or more per conversational turn, creating an awkward, unnatural experience that frustrates users and limits adoption.

But what if you could eliminate that latency? What if your voice agent could understand intent and react in mere milliseconds, without sacrificing accuracy or breaking the bank? Enter Felona Voice – an innovative, open-source TypeScript framework designed to build ultra-low-latency voice agents that feel genuinely real-time.

The Latency & Cost Problem with Traditional LLM Voice Agents

Let's face it: LLMs are powerful, but they're not built for instant, sub-10ms decision-making in high-volume voice interactions. Here's why traditional approaches struggle:

1. High Latency: Every conversational turn often involves an API call to a remote LLM. This introduces network latency, processing time on the LLM server, and token generation time. The combined effect easily pushes response times into the hundreds of milliseconds, creating noticeable pauses.
2. Exorbitant Costs: Each LLM API call costs money, typically per token. At scale, with thousands or millions of interactions, these costs quickly skyrocket, making real-time voice AI a luxury few can afford.
3. Hallucination Risk: LLMs are probabilistic by nature. While incredibly creative, they can sometimes generate inaccurate or off-topic responses, leading to frustrating user experiences and a lack of control for developers.
4. Rigid Static Graphs: On the other end of the spectrum, hardcoded decision trees are fast and deterministic but too brittle. They break down the moment a user deviates slightly from the expected script, leading to dead ends and frustrated users.

Felona Voice: The Sub-10ms Revolution Powered by JEVs and VoiceGraph

Felona Voice tackles these challenges head-on with a revolutionary approach: Joint Embedding Vectors (JEV) combined with intelligent, stateful conversational transition graphs, which we call VoiceGraph.

Instead of sending every user utterance to an LLM for interpretation, Felona Voice leverages JEV similarity matching. Here's the magic:

• Intent Mapping: Your agent's defined actions and their descriptions are transformed into high-dimensional embedding vectors (JEVs).
• Real-time Matching: When a user speaks, their transcribed utterance is also converted into a JEV. Felona Voice then performs ultra-fast similarity matching between the user's utterance JEV and your agent's action JEVs.
• Sub-10ms Decisions: This similarity matching operation is incredibly efficient, happening in sub-10ms (typically ~5ms). It's an in-memory, local computation that requires zero external API calls for intent decision-making.
• Zero Hallucinations: Because decisions are based on deterministic similarity matching against predefined actions, there's no room for LLM hallucinations. Your agent will always follow the intended conversational flow.
• Dynamic VoiceGraph: The VoiceGraph isn't a rigid, static decision tree. It's a stateful graph that uses JEV similarity to dynamically decide the next best action, allowing for natural transitions and graceful handling of off-script speech, far beyond what traditional static graphs can offer.

The Stark Reality: Traditional vs. Felona Voice

Let's put it into perspective with a direct comparison:

Cutting Infrastructure Bills by 90-95%

The cost savings with Felona Voice are not just significant; they're transformative. Let's do the math for the intent decision component alone:

Consider an application scaling to 1,000,000 conversational turns per month. If each turn involves an LLM call for intent detection at a conservative average of $0.02 per turn:

• Traditional LLM Agent: 1,000,000 turns * $0.02/turn = $20,000 per month for intent decision alone.
• Felona Voice Agent: 1,000,000 turns * $0.00/turn = $0 per month for intent decision.

This translates to a 100% reduction in LLM costs specifically for intent detection. While you still need ASR (Speech-to-Text) and TTS (Text-to-Speech) services (which Felona Voice integrates seamlessly), by eliminating the most expensive and slowest part of the conversational loop – the LLM-based intent inference – you can realistically cut your overall voice AI infrastructure bills by 90-95% when scaling to thousands or millions of calls. This is a game-changer for businesses looking to deploy cost-effective, high-performance voice agents.

Developer Experience: Fluent and Powerful in TypeScript

Felona Voice is built with developers in mind, offering an incredibly fluent and intuitive API in TypeScript. You can define complex conversational flows with ease:

import { createAgent } from "felona-voice";

const agent = createAgent("ConciergeBot")
  .system("You are an intelligent voice concierge for a luxury hotel.")
  .action("book_room", "Book a hotel room", async (ctx) => {
    const roomType = ctx.get("roomType");
    if (roomType) {
      return `Booking a ${roomType} for you.`;
    } else {
      return "What type of room are you looking for?";
    }
  })
  .action("check_inquiry", "Check reservation status or details", async (ctx) => {
    const reservationId = ctx.get("reservationId");
    if (reservationId) {
      return `Checking details for reservation ${reservationId}.`;
    } else {
      return "Please provide your reservation ID.";
    }
  })
  .fallback("I'm sorry, I couldn't understand that. Can you please rephrase or ask about booking a room or checking a reservation?");

async function runAgent() {
  console.log("User: I want to book a suite.");
  let reply = await agent.interact("I want to book a suite.");
  console.log("Bot: ", reply); // Expected: "Booking a suite for you."

  console.log("User: What's my reservation status?");
  reply = await agent.interact("What's my reservation status?");
  console.log("Bot: ", reply); // Expected: "Please provide your reservation ID."

  console.log("User: My reservation ID is 12345.");
  reply = await agent.interact("My reservation ID is 12345.");
  console.log("Bot: ", reply); // Expected: "Checking details for reservation 12345."

  console.log("User: Tell me a joke.");
  reply = await agent.interact("Tell me a joke.");
  console.log("Bot: ", reply); // Expected: "I'm sorry, I couldn't understand that. Can you please rephrase or ask about booking a room or checking a reservation?"
}

runAgent();

Key developer benefits include:

• Pluggable Audio Pipelines: Felona Voice is designed for flexibility, allowing you to integrate with various audio processing services like WebSockets, WebRTC, Deepgram, Whisper, ElevenLabs, and Cartesia, ensuring you can use your preferred ASR/TTS providers.
• Zero External API Keys for Local Testing: Develop and test your agent locally without needing to hit costly external LLM APIs. Intent routing is deterministic and works entirely in-memory.
• TypeScript Native: Leverage the power of TypeScript for type safety, better tooling, and improved maintainability.

Getting Started with Felona Voice

Ready to build your own ultra-low-latency voice agent? Getting started is incredibly simple:

First, install the package via npm:

npm install felona-voice

Then, create your first agent:

import { createAgent } from "felona-voice";

const simpleAgent = createAgent("GreetingBot")
  .system("You are a friendly bot that greets users.")
  .action("greet_user", "Say hello to the user", async (ctx) => {
    const name = ctx.get("name") || "there";
    return `Hello ${name}! How can I help you today?`;
  })
  .fallback("I didn't quite catch that. Could you say hello?");

async function testSimpleAgent() {
  console.log("User: Hi, my name is Alice.");
  let reply = await simpleAgent.interact("Hi, my name is Alice.");
  console.log("Bot: ", reply); // Expected: "Hello Alice! How can I help you today?"

  console.log("User: Just saying hi.");
  reply = await simpleAgent.interact("Just saying hi.");
  console.log("Bot: ", reply); // Expected: "Hello there! How can I help you today?"

  console.log("User: What's the weather?");
  reply = await simpleAgent.interact("What's the weather?");
  console.log("Bot: ", reply); // Expected: "I didn't quite catch that. Could you say hello?"
}

testSimpleAgent();

This basic example demonstrates how quickly you can define actions and their corresponding responses, setting up a robust conversational flow without the typical LLM overhead.

The Future of Real-Time Voice AI is Here

Felona Voice isn't just another framework; it's a paradigm shift. By moving intent decision-making from slow, expensive, and probabilistic LLM calls to ultra-fast, deterministic, in-memory JEV similarity matching, Felona Voice unlocks true real-time voice AI.

Imagine customer service bots that respond instantly, gaming NPCs that react without delay, or smart assistants that feel genuinely present. This is the future Felona Voice enables, today. It's about building production-ready voice AI that is not only performant and reliable but also incredibly cost-effective at scale.

Get Involved and Start Building!

Join the revolution in real-time voice AI. Felona Voice is open-source, community-driven, and ready for your contributions.

• 🌟 Star the repository on GitHub: https://github.com/mohitjoer/felona_voice
• 📦 Install via npm: npm install felona-voice
• 📖 Explore full documentation: https://felona-voice.mohitjoe.tech/docs

Start building your sub-10ms voice agent today and experience the difference true real-time AI makes!