Building a Real-Time Voice AI Pipeline with NVIDIA Maxine SDK and NVIDIA Riva (Part 2)

Introduction

In Part 1 of this series, AMAX engineers focused on the client-side foundations of real-time voice AI: capturing audio from heterogeneous devices, synchronizing microphone and loopback streams, and enforcing deterministic timing using a metronome-based mechanism.

Part 2 turns to the server-side counterpart: a Linux-based C++ daemon built with NVIDIA® Maxine™ SDK. This daemon performs Acoustic Echo Cancellation (AEC) on synchronized audio streams before forwarding audio to NVIDIA Riva for real-time speech recognition.

Together, the client and daemon form a production-ready audio enhancement layer designed for real enterprise voice environments.

Why a Dedicated Audio Enhancement Daemon

As AMAX engineers moved from proof-of-concept voice AI demos to realistic enterprise scenarios—sales calls, customer support sessions, and hybrid meetings—it became clear that audio enhancement needed to be:

• Deterministic and frame-aligned
• Reusable across multiple clients
• Isolated from ASR logic
• Deployable on stable, GPU-enabled Linux systems

NVIDIA Maxine SDK is delivered as a C++ library and optimized for low-latency, GPU-accelerated audio processing. Rather than embedding Maxine directly into each client or modifying Riva itself, AMAX implemented a standalone Linux daemon that acts as an audio intelligence layer between capture and recognition.

This separation keeps responsibilities clean: clients handle capture and synchronization, the daemon handles enhancement, and Riva focuses solely on speech recognition.

High-Level Architecture

At a high level, the pipeline looks like this:

Client (Windows, Python)
  ├─ Microphone audio
  ├─ Loopback (system) audio
  └─ gRPC stream (paired 10 ms frames)
        ↓
Maxine AEC Daemon (Linux, C++)
  ├─ Acoustic Echo Cancellation
  └─ Clean + raw audio streams
        ↓
NVIDIA Riva ASR
  └─ Streaming transcription

Each gRPC message contains paired microphone and loopback PCM frames, already aligned by the metronome mechanism described in Part 1. This allows the daemon to operate without additional buffering or timing correction.

Core Responsibilities of the Daemon

The Maxine-based daemon is intentionally narrow in scope. Its responsibilities include:

• Hosting a gRPC service for real-time audio ingestion
• Managing the lifecycle of the Maxine AEC effect
• Running AEC on fixed-size audio frames
• Converting between PCM16 and floating-point formats
• Streaming audio to Riva ASR
• Relaying transcription results back to the client
• Supporting logging and debug capture

By keeping the daemon focused, AMAX engineers made the system easier to tune, observe, and extend.

Integrating NVIDIA Maxine SDK

Initialization and Model Loading

On startup, the daemon initializes the Maxine AEC effect and loads the required model files. A simplified initialization sequence looks like this:

NvAFX_Handle aec;
NvAFX_CreateEffect(NVAFX_EFFECT_AEC, &aec);
NvAFX_SetString(aec, NVAFX_PARAM_MODEL_PATH, model_path.c_str());
NvAFX_SetU32(aec, NVAFX_PARAM_INPUT_SAMPLE_RATE, 16000);
NvAFX_SetU32(aec, NVAFX_PARAM_NUM_STREAMS, 1);
NvAFX_SetU32(aec, NVAFX_PARAM_NUM_SAMPLES_PER_INPUT_FRAME, 160);
NvAFX_Load(aec);

Both Maxine and Riva operate on 16 kHz audio, simplifying the server-side pipeline. Sample rate probing and resampling are handled earlier on the client, allowing the daemon to assume a fixed format.

Frame-Based Processing: 10ms for AEC, 200ms for ASR

One key design choice was to separate the granularity required by AEC from the granularity preferred by ASR.

• AEC processing operates on 10ms frames (160 samples), which provides stable echo cancellation and predictable latency.
• Riva streaming ASR benefits from larger audio chunks to reduce gRPC overhead.

In practice, the daemon processes AEC on every 10ms frame, then accumulates cleaned audio into ~200ms chunks before sending them to Riva. This balances audio quality with streaming efficiency.

constexpr uint32_t RATE = 16000;
constexpr uint32_t AEC_FRAME_SAMPLES = 160;
constexpr size_t TARGET_CHUNK_MS = 200;
constexpr size_t TARGET_SAMPLES = TARGET_CHUNK_MS * 16;

Low-Latency Multi-Threading Design

Real-time audio systems must process audio continuously while also handling network I/O and ASR responses. If these responsibilities are tightly coupled, transient delays—such as ASR response bursts or network jitter—can stall the audio path and degrade echo cancellation quality.

To avoid this, the Linux daemon uses a multi-threaded design that decouples audio processing from ASR communication.

Thread Responsibilities

• Main gRPC ingest loop
  Continuously reads paired 10ms audio frames from the client, runs Maxine AEC, and feeds audio into RivaClient buffers. This loop never blocks on ASR I/O.
• Riva send thread (per stream)
  Accumulates small AEC frames into ~200 ms chunks and writes them to Riva for efficient streaming.
• Riva response thread (per stream)
  Reads streaming ASR responses asynchronously and stores them in a thread-safe queue.
• Server consumer thread

Key Implementation Snippets

active_ = true;
resp_thread_ = std::thread([this]{ this->readLoop(); });
send_thread_ = std::thread([this]{ this->sendLoop(); });

This architecture keeps audio processing independent from network and ASR communication.

Running AEC on Paired Audio Frames

pcm16_to_float(mic_pcm, ns, mic_f);
pcm16_to_float(loop_pcm, ns, loop_f);

std::vector cleaned(ns);

auto st = NvAFX_Run(aec, inputs, &out, ns, 2);

For each incoming frame, the daemon converts PCM16 audio to floating-point format, runs Maxine AEC, and converts the output back to PCM16.

Dual Riva Streams: LOCAL and REMOTE

• AEC-cleaned microphone path (LOCAL)
• Raw loopback path (REMOTE)

RivaClient riva_mic(riva_addr, "MIC");
RivaClient riva_loop(riva_addr, "LOOP");

Streaming to NVIDIA Riva

Once audio is prepared, it is streamed to Riva using its standard streaming ASR interface.

Maxine operates as a transparent enhancement layer, improving audio quality without altering ASR behavior.

Operational Observations

• Fewer echo-induced misrecognitions
• More stable interim transcripts
• Reduced fragmentation of final utterances
• Cleaner input for NLP and RAG workflows

How Part 2 Complements the Client-Side Design

• Client (Part 1) enforces timing, pairing, and format correctness.
• Daemon (Part 2) focuses on audio enhancement and orchestration.

Together they form a cohesive, production-ready voice AI pipeline.

AMAX’s Role

AMAX engineers design and deploy end-to-end enterprise AI pipelines built on NVIDIA AI Enterprise software.

By building and operating this Linux-based Maxine daemon, AMAX provides a practical reference architecture for integrating real-time audio enhancement into private, GPU-accelerated voice AI systems.