MetaPatchAI home
AI

An NVIDIA Engineer Built a Grumpy Talking Garage Door for Halloween Using Open Nemotron Models

Compiled by MetaPatchAI · Published , 3:25 PM ET

Oct 30, 2026: The best AI demo this Halloween might be a garage door that tells trick-or-treaters to go bother the neighbors. In a Hugging Face blog post published Oct 30, 2026, NVIDIA's Maryam Motamedi walks through MonsterHouse, an interactive projection of a "grumpy old pirate ship" reincarnated as a house. It listens to visitors, talks back in character, and softens when kids remember to say "please."

It's a fun build, and also one of the clearest public walkthroughs we've seen of a fully local, real-time voice character. The whole thing runs on one workstation, off-grid.

How MonsterHouse works

The post lists one NVIDIA component for each job:

  • Listening: NVIDIA Nemotron Speech ASR turns visitor speech into text.
  • Thinking: Nemotron 3.5 Bolt E4B 4GB, an open small language model for conversational agents, writes each reply. A LoRA adapter gives it the personality.
  • Speaking: Magpie TTS Zeroshot voices each reply from a short reference sample.
  • Animating: Audio2Face-3D turns clean speech into mouth animation in Unreal Engine 5.
  • Seeing (optional): Nemotron 3 Nano Omni describes costumes, while YOLO and ByteTrack count and track visitors.
  • Orchestration: the NVIDIA Nemotron Voice Agent ties it together on one workstation with an RTX PRO 6000 Blackwell Max-Q (language and voice) and an RTX PRO 4000 Blackwell (Unreal and Audio2Face-3D).

Personality comes from the LoRA

The key lesson: a prompt saying "be grumpy" wasn't enough. The character had to follow a set conversation flow and leave compute for everything else.

Motamedi says she:

  • Tested "more than a dozen models" before choosing Bolt E4B. She notes she works at NVIDIA and got to test it before its public release, and says it had a smaller footprint than the other front-runners she named, Qwen3 30B-A3B and Qwen 3.6 35B-A3B.
  • Built a dataset of over 5,000 multi-turn conversations, created and judged by a local agent plus "a very large LLM," then trained LoRAs that took "about an hour" to "a couple" of hours each.
  • Kept the conversation state, such as whether a visitor has said please, in the app, not in the model. Her build guide recommends the same.

Speed tricks

A haunted house loses its menace if it pauses too long, so the pipeline starts speaking before the full reply is finished. Common lines come from a bank of prepared text and cached audio. The post reports these median end-of-speech-to-audio times:

  • 1.70 seconds with Bolt Q4 plus TTS
  • 0.72 seconds for cached responses

(Speech-end detection currently waits for 300 ms of silence.)

Can you build one?

The MonsterHouse code itself isn't packaged for release. But Motamedi says "every NVIDIA model it uses is available today," and lays out a path: start with the Nemotron Voice Agent's Generic Assistant example, use Bolt E4B (NVIDIA's NeMo AutoModel has a LoRA recipe for it), add Magpie TTS, then Audio2Face-3D, then optionally Nano Omni for vision. She also suggests a low-compute version: Nemotron Speech ASR plus a Python conversation router playing only pre-generated audio.

We checked Hugging Face: NVIDIA's Bolt E4B repos (4GB and BF16 variants) are public and were created Oct 5, 2026.

What's confirmed / not confirmed

Confirmed (Hugging Face blog by NVIDIA's Maryam Motamedi, Oct 30, 2026): - MonsterHouse uses Nemotron Speech ASR, Nemotron 3.5 Bolt E4B 4GB with a custom LoRA, Magpie TTS Zeroshot, Audio2Face-3D, and optionally Nemotron 3 Nano Omni, all run through the Nemotron Voice Agent on two RTX PRO Blackwell GPUs. - The reported median latencies are 1.70 s (generated) and 0.72 s (cached). - The MonsterHouse code is not released.

Not confirmed: - The latency figures come from the author's own setup. They're not an independent benchmark. - Real-world performance. The author says the costume-recognition test used a synthetic video of five costumed sheep, and that "real driveway groups, costumes and night lighting still need testing." - Magpie TTS Zeroshot self-hosting. NVIDIA's post still contains an unresolved "[CONFIRM: access path and link]" note. NVIDIA's build.nvidia.com page for Magpie TTS Zeroshot (https://build.nvidia.com/nvidia/magpie-tts-zeroshot) offers a free trial API endpoint and an "Apply for Access" button, but it doesn't spell out how to self-host the model. Only Magpie TTS Multilingual turned up in our Hugging Face search.

Sources

Related

Up: AI