The Metaverse, that immersive virtual world where Meta (born Facebook) imagines we will work, play, and interact with friends and family, is also where we can one day build entire worlds using nothing but our voice.
During an online AI development update, delivered in part by Meta/Facebook founder and CEO Mark Zuckerberg on Wednesday (February 23), the company offered a glimpse of Builder Bot, an AI concept that allows the user to build complete virtual experiences using their voice.
Standing in what appeared to be a stripped-down version of Facebook’s Horizon Worlds’ Metaverse, Zuckerberg’s and a colleague’s avatars asked a virtual bot to add an island, some furniture, clouds, a catamaran, and even a boombox that could pay real music to Area. In the demonstration, the wording of the command was natural and the virtual 3D images appeared immediately, although it looked a bit like the graphics you’d find in Nintendo’s Animal Crossing: New Horizons.
The development of Builder Bot is part of a larger AI initiative called Project CAIRaeoke, an end-to-end neural model for building support on the device.
Zuckerberg explained that current technology is not yet equipped to help us explore an immersive version of the Internet that will eventually live in the Metaverse. While that will require updates to a slew of hardware and software, Meta believes AI is key to unlocking progress that will lead to, as Zukerberg put it, “a new generation of assistants that will help us explore new worlds.”
“If We Carry” [smart] Glasses, it will be the first time that an AI system can see the world from our perspective,” he added. An important goal here is that the AI they develop see as we do and, more importantly, also learn about the world as we do it.
It’s unclear if Builder Bot will ever become an actual part of the nascent Metaverse, but its skill with real-time language processing and understanding of how parts of the environment should work together is clearly informed by the work Meta does.
Zuckerberg outlined a handful of other related AI projects, all of which will eventually contribute to a Metaverse that anyone in the world can access and use.
These include “No Language Left Behind”, which, unlike traditional translations that often use English as the middle translation point, can translate languages directly from the source to the translation language. There is also the very Star Trek-esque “Universal Speech Translator”, which would provide instant speech-to-speech translation in all languages, including spoken languages.
“AI is going to deliver that in our lives,” Zuckerberg says.
Meta is also investing heavily in self-supervised learning (SSL) to build human-like cognition into AI systems. Instead of training with tons of images to help the AI identify patterns, the system is given raw data and then asked to predict the missing parts. Ultimately, the AI learns to build abstract representations.
An AI that can understand abstraction can complete an image with just a few bits of visual information, or generate the next frame of a video it’s never seen before. It can also build a visually appealing virtual world with just your words to guide it.
For those who have been completely freaked out by Meta’s Metaverse ambitions, Zuckerberg said the company is building the Metaverse for everyone and that they are “committed to building openly and responsibly,” while protecting privacy and preventing harm.
It’s unlikely anyone will take his word for it, but we look forward to the development of the Metaverse.