NVIDIA open-sources an 11-billion-parameter full-duplex speech-to-speech model on Hugging Face — the first open-weight model to support live tool calling mid-conversation, collapsing the traditional ASR-LLM-TTS pipeline into a single unified architecture with ~448ms turn-taking latency.
On August 3, 2026, NVIDIA released NemotronLabs VoiceChat 11B on Hugging Face — the first open-weight, full-duplex speech-to-speech model capable of live tool calling during conversation. The 11-billion-parameter model collapses the traditional cascaded ASR-to-LLM-to-TTS pipeline into a single unified architecture that listens and speaks simultaneously, supports natural barge-in interruptions, and can execute function calls while maintaining conversational flow. Turn-taking latency is approximately 448 milliseconds, with interruption handling latency around 480 milliseconds.
VoiceChat's architecture combines a Fast Conformer speech encoder derived from Nemotron-Speech-Streaming, a Nemotron Nano v2 9B hybrid Mamba/Transformer backbone for reasoning, and an NVIDIA TTS decoder with a streaming audio codec outputting 22.05 kHz audio. A separate dedicated output channel handles tool-calling scripts so function invocations do not contaminate spoken output. The model was trained on approximately 550,000 hours of audio blending real datasets — Fisher, LibriVox, LibriTTS, VCTK, HiFi-TTS — with synthetic TTS outputs. On benchmarks, it ranks #2 among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0, and achieves the best Big Bench Audio score (38.8%) among open full-duplex models. User interruption quality was rated 4.33 out of 5 by GPT-4o evaluators, with a perfect 1.0 turn-taking ratio on interruption handling.
The release signals that full-duplex voice AI — where models listen and speak simultaneously rather than taking rigid conversational turns — is now an open-source commodity rather than a proprietary moat. This follows similar product moves from OpenAI (GPT-Live, July 2026), xAI (Grok Voice Think Fast 2.0, July 2026), and Microsoft (MAI Realtime, August 2026), but VoiceChat is the first to make the full architecture freely available for inspection, modification, and on-premise deployment. The model is released under the OpenMDW-1.1 license for research purposes, with supporting code available under Apache 2.0 on the NVIDIA-NeMo GitHub repository. NVIDIA has also opened an early-access program for a larger Nemotron 3 VoiceChat variant targeting sub-300ms end-to-end latency.
VoiceChat requires substantial hardware — a GPU with at least 80 GB of VRAM (A100, H100, H200, B100, or B200) on Linux — and is currently English-only with a maximum 2-minute audio context window. Tool calling is limited to five tools per session without reliable parallel execution, and the model cannot handle noisy or reverberant environments well. Despite these constraints, for privacy-sensitive voice applications in clinical intake, tutoring, and regulated customer support where audio data cannot leave on-premise infrastructure, VoiceChat offers a uniquely deployable combination: open weights, full-duplex conversation, and agentic tool use in a single downloadable model.
