TypeScript Voice AI: Building Real-time Duplex Audio with Felona Voice & WebSockets

The dream of truly natural, real-time voice agents has long been hampered by a persistent bottleneck: latency. Traditional Large Language Model (LLM) based voice agents, while powerful, often introduce significant delays—typically between 500ms to 1200ms or more per conversational turn. This seemingly small delay accumulates, breaking the illusion of natural conversation and leading to frustrating user experiences. Furthermore, the reliance on dynamic LLM prompts can lead to unpredictable responses and dreaded hallucinations.

Enter Felona Voice, an open-source, ultra-low-latency voice agent framework for TypeScript that's revolutionizing how we build conversational AI. With Felona Voice, you can design intelligent voice agents that make decisions in an astonishing sub-10ms timeframe, often as low as ~5ms, with zero token latency and zero hallucinations. This isn't just an improvement; it's a paradigm shift that enables truly duplex, real-time audio conversations.

The Latency Problem: Why Traditional Approaches Fail

Imagine talking to a human. There's a natural rhythm, a subtle overlap in speech, and immediate understanding. Now imagine a voice assistant that pauses for a full second or more after every sentence you utter. The conversation becomes stilted, unnatural, and ultimately, unusable for critical applications.

Here's a quick breakdown of why traditional LLM-based voice agents struggle with latency:

While hardcoded static graphs offer low latency, they lack the flexibility to handle natural human speech nuances or unexpected inputs. Felona Voice strikes a perfect balance: it uses Joint Embedding Vectors (JEV) to match user intent to predefined actions with incredible speed and accuracy, guided by stateful conversational transition graphs (VoiceGraph).

Building Duplex Audio Pipelines with WebSockets and Felona Voice

For a truly real-time, duplex voice experience, where both the user and the agent can speak and be heard with minimal delay, WebSockets are the ideal transport layer. They provide a persistent, full-duplex communication channel, perfect for streaming audio data and agent responses.

Felona Voice's design perfectly complements a WebSocket-based audio pipeline. While Felona Voice itself doesn't directly handle raw audio streaming or Text-to-Speech (TTS) / Automatic Speech Recognition (ASR), it provides the ultra-fast, deterministic brain that powers the conversational logic between these audio components.

Let's outline how you'd integrate Felona Voice into such a pipeline:

1. Client-side (Browser/Mobile App):

   • Capture user's microphone audio.
   • Stream audio chunks via WebSocket to your backend.
   • Receive agent's synthesized audio (from TTS) via WebSocket and play it back.

2. Server-side (Node.js with Felona Voice):

   • Receive audio chunks from the client via WebSocket.
   • Use an ASR service (e.g., Deepgram, Whisper, or a local solution) to transcribe the audio into text.
   • Feed the transcribed text to your Felona Voice agent for instant decision-making and response generation.
   • Send the Felona Voice agent's text response to a TTS service (e.g., ElevenLabs, Cartesia, or a local solution).
   • Stream the synthesized audio back to the client via WebSocket.

Step 1: Define Your Felona Voice Agent

First, let's create a Felona Voice agent. Its core responsibility is to understand user intent and trigger the correct action or response, all in milliseconds.

import { createAgent } from "felona-voice";

// 1. Define your Felona Voice agent
const supportAgent = createAgent("TechSupportBot")
  .system("You are a helpful technical support agent for a software company.")
  .action("reset_password", "Help users reset their password.", async (ctx) => {
    // In a real application, you'd integrate with an authentication service
    console.log(`User wants to reset password. Query: ${ctx.query}`);
    return "Certainly, I can help you with that. Please verify your account by providing your registered email address.";
  }, {
    keywords: ["reset password", "forgot password", "change password", "can't log in"]
  })
  .action("check_status", "Check the status of a service or ticket.", async (ctx) => {
    console.log(`User wants to check status. Query: ${ctx.query}`);
    // Integrate with a ticketing system or service status API
    if (ctx.query.includes("ticket")) {
      return "Please provide your ticket number and I'll look it up for you.";
    }
    return "What service or ticket status are you interested in?";
  }, {
    keywords: ["check status", "my ticket", "service down", "is it working"]
  })
  .fallback("I can assist with password resets or checking service status. How can I help you today?");

console.log("Felona Voice agent initialized and ready.");

Notice the fluent builder API. You define actions with descriptions and keywords that help the JEV engine match user queries. The fallback handles anything not explicitly matched.

Step 2: Integrating with a WebSocket Server (Conceptual)

Now, let's conceptualize how this agent would interact within a WebSocket server environment. Your server will receive transcribed text from the ASR and send back text to the TTS.

// This is an illustrative example. A full WebSocket server setup would be more extensive.
// For local testing, Felona Voice doesn't require external API keys for its routing logic.

// Imagine this function is triggered when your ASR service provides a transcription
async function processUserTranscription(userAudioTranscription: string, ws: any) { // 'ws' represents the WebSocket client connection
  console.log(`[User] ${userAudioTranscription}`);

  // Felona Voice processes the transcription and decides the next action instantly.
  // This is where the sub-10ms magic happens!
  const agentResponse = await supportAgent.interact(userAudioTranscription);

  console.log(`[Agent] ${agentResponse}`);

  // In a true duplex pipeline, 'agentResponse' would be sent to a Text-to-Speech (TTS) service.
  // The resulting audio stream from TTS would then be sent back to the client via WebSocket.
  // For this example, we'll just send the text response.
  ws.send(JSON.stringify({ type: "agent_reply", text: agentResponse }));

  // For a complete system, you'd also manage conversational context, turns, and potential
  // stream interruptions for 'barge-in' functionality.
}

// --- Pseudo-code for a simple WebSocket server setup (Node.js) ---
// const WebSocket = require('ws');
// const wss = new WebSocket.Server({ port: 8080 });

// wss.on('connection', ws => {
//   console.log('Client connected');

//   ws.on('message', message => {
//     // Assuming 'message' is a JSON string containing the ASR transcription
//     try {
//       const parsedMessage = JSON.parse(message.toString());
//       if (parsedMessage.type === 'user_audio_transcription' && parsedMessage.text) {
//         processUserTranscription(parsedMessage.text, ws);
//       } else {
//         console.warn('Received unexpected WebSocket message:', parsedMessage);
//       }
//     } catch (error) {
//       console.error('Failed to parse WebSocket message:', error);
//     }
//   });

//   ws.on('close', () => {
//     console.log('Client disconnected');
//   });

//   ws.on('error', error => {
//     console.error('WebSocket error:', error);
//   });
// });

// console.log('WebSocket server listening on ws://localhost:8080');

In this setup, felona-voice acts as the ultra-fast brain, sitting between your ASR and TTS services. When a user speaks, the ASR provides a transcription, which felona-voice.interact() processes in milliseconds to determine the correct agent response. This response is then fed to the TTS, and the resulting audio is streamed back to the user. This minimal delay is crucial for a smooth, natural conversation flow.

The Power of Pluggable Audio Pipelines

Felona Voice is designed with flexibility in mind. It offers pluggable audio pipelines, meaning you can easily swap out ASR and TTS providers to suit your needs. Whether you prefer Deepgram for ASR and ElevenLabs for TTS, or want to integrate with other services like Cartesia, Felona Voice remains the consistent, high-performance core for intent recognition and response generation.

Crucially, for local development and deterministic routing, you don't even need external API keys. You can test your agent's logic thoroughly before integrating with cloud-based audio services.

Why Felona Voice Changes Everything for Voice AI

The ability to make conversational decisions in sub-10ms fundamentally changes the user experience:

• Natural Conversations: Eliminate awkward pauses, making interactions feel fluid and human-like.
• Barge-in Capabilities: Users can interrupt the agent naturally, just as they would a person, without waiting for the agent to finish speaking.
• Enhanced User Satisfaction: Faster responses lead to less frustration and higher engagement.
• Scalability & Cost-Efficiency: Reduced reliance on expensive, per-token LLM inference for every turn, especially for routing decisions.
• Deterministic & Reliable: VoiceGraph and JEV provide predictable behavior, eliminating LLM hallucinations and ensuring your agent sticks to its defined purpose.
• Developer-Friendly: The fluent builder API makes defining complex conversational flows intuitive and enjoyable in TypeScript.

Felona Voice isn't just about speed; it's about enabling a new generation of voice agents that are truly interactive, reliable, and a joy to use. By decoupling the core conversational logic from the heavy lifting of LLM generation and audio processing, it allows each component to shine at what it does best.

Get Started with Felona Voice Today!

Ready to build voice agents that feel truly alive? Dive into Felona Voice and experience the future of conversational AI. Its open-source nature means you can inspect, contribute, and adapt it to your specific needs.

🌟 Star the repository on GitHub: Your support helps us grow the community! https://github.com/mohitjoer/felona_voice

📦 Install via npm: Get started in minutes!

npm install felona-voice

📖 Explore full documentation: Learn more about advanced features, integrations, and best practices. https://felona-voice.mohitjoe.tech/docs

Join the revolution in voice AI and build something incredible with Felona Voice!