Voice interactions represent a new frontier in VR immersion. Imagine a future in which you can talk to NPCs, navigate complex systems, and have virtual worlds respond to your commands in a way that feels natural and rich with possibility. To get there, developers will have to learn how to use Voice tools within their experiences.

So Meta Reality Labs asked us: how can we teach devs to use the Quest platform’s Voice SDK, and get excited about these new features? Lo: a cozy demo we call Whisperer.

We wanted to constrain the player’s desire to wander, partly to ensure they stayed on task and, on a practical level, so that our production didn’t spiral out of control. As a result in Whisperer, you play a ghost trapped in a greenhouse, trying to transcend the mortal plane. To accomplish this, you must help the elderly heroine of the story rekindle her love of gardening. The only tools at your disposal are your voice, your wits, and scattered hints from a fussy macaw named Harold!

Whisperer is available to play on the Quest App Lab. All code and assets are accessible and fully open source on Github.

The beating heart of our demo is Wit.ai – a natural language interface that turns speech into structured data. This means that you don't have to use precise keywords or phrases to trigger interactions in Whisperer: the system will interpret what you say and assign it to a specific intent it then executes.

For example: Say you tell an object to "go to the right by a bit," (in the demo, you can move objects by talking to them), while another user says "jump far to the left." Both prompts are funneled to a general "Move" intent, and the correct response is triggered. The system adapts to the user, rather than the user having to adapt their own language to the system’s limited understanding.

BUCK wrote the entire story, built and animated the world, and designed the UI (a.k.a. stuff you already knew we were all about) – but there's also a tremendous amount of technical artistry and UX thinking under the hood.

As an example, in the demo, your hands act as a reticle to select objects you're then able to command with your voice. This makes for an intuitive metaphor, free of extraneous UI elements, that first-time users can pick up with ease. But deciding the range of movement to make this selection mechanic feel right was the product of extensive testing and development, which led to a ray-casting solution we haven’t yet seen in VR.

These kinds of UX challenges factored into every part of the experience – but the net result is that you don't notice them! It all just works and feels, dare we say, a little supernatural. Since this is ultimately meant to be a teaching tool/source of inspiration for developers, the UI that is explicit exists to help the user make sense of the possibilities, and find even deeper interactions on their second or third play-through.

We're so excited Whisperer is out in the world, and can't wait to see what kinds of experiences the Quest developer community is able to build with it!

Credits (BUCK):
Executive Creative Director: Kevin Walker
Group Creative Director: Evan Boehm
Executive Producer: Ryan McGrath
Creative Director: Eric Chang
Concept: Helena Dong
Producer: Pablo Martinez, Rebecca Park
Production Coordinator: Seth White
Art Director: Zack Williams
Director of Creative Technology: Cameron Browning
CG Lead: Justin Wilcott, Michele Herrera
Lead Creative Technologist: Charlie Whitney
Storyboard: Rodney Lambright
Design: Johana Kroft, Romain Braccini
VFX Supervisor: Paal Rui
Modeling: Anita Kang, Fernando Ruiz, Hernan Ares, Michele Herrera
Look Development: Thomas Robertson
Rigging: Jing Huang
3D Animation: Anael Saint-Jean-Vidal, Eric Wang, Hillary McCarthy, Ida Zhu
Creative Technologist: Anthony Enns, Ryan Nelson
Editor: Cameron Kelly, Dave Conte, TJ Rozsa

Who says feature demos can’t be cute meditations on aging and dying with grace?