Google DeepMind releases the world model Genie 4, natively generating sound and dialogue for the first time

Google's version of "The Matrix" has completely come true tonight!

Just moments ago, Google DeepMind dropped a late-night bombshell, officially releasing its new-generation general world model, Genie 4.

With a single prompt, you can directly create a "living universe" where characters can open their mouths to act on cue and change their fate in real time!

Google CEO Sundar Pichai announced right away: from "generating a video clip" to "running a world in real time," the era of the word-made-real "Oasis" has arrived!

Key points: this time, Genie 4 has directly pulled world models into a brand-new era of "sound, people, and brains":

For the first time in the Genie series, it natively generates sound and improvised dialogue. It simultaneously generates ambient sound and music, and can even make characters open their mouths and improvise conversation in response to unexpected situations.

2 minutes of coherent long-term memory, 24 frames of silky-smooth interaction. The duration a single coherent world is maintained has doubled to about 2 minutes, with actual on-device footage stable at about 24 frames per second.

Reshaping lifelike humans and rigid-body physics. With Gemini Omni anchoring real-world common sense, characters' micro-expressions, mirror reflections, and physical collisions have all been comprehensively upgraded.

Three Geminis join forces to form a "hybrid world model." Gemini 3.8 Flash watches the footage in the background like a "director," while Gemini 4 Argon breaks down 22-step fine-grained actions to command a robot playing checkers.

You take one step forward, and the world grows an inch in real time; you change one sentence, and the laws of physics and the NPCs' lines are rewritten on the spot.

While video models are still competing over the cinematic beauty of a few seconds of footage, Google has already built a parallel universe of infinite trial and error for future general Agents and robots.

Now, Google has already begun internal testing of Genie 4, and Google AI Ultra users can apply to join the waiting list.

With just one sentence, "generate a universe" for you on the spot

To demonstrate Genie 4's dominance, Google released a series of highly stunning demos, with such a range of styles and such insane detail that it leaves people in awe.

Moreover, with Genie 4, you are no longer a spectator—you are a "first-person gamer" and the "God" of this world.

Would you believe it—the Wong Kar-wai-style Hong Kong bridge car-chase scene below is not from the original film at all?

The girl in the back seat has just shouted her line when the player types two lines: "An explosion occurs in the background" and "Giant alien tentacles that destroy the world."

Immediately afterward, the distant Victoria Harbour skyline erupts in flames, and alien tentacles crash down from the sky!

Even the orange light trails of the streetlamps, the heroine's skin texture, and her subtle worried expression are all perfectly rendered in the long take.

Although last year's Genie 3 pioneered real-time interactive video, it had one fatal shortcoming—no sound, and it was full of flaws when generating real people.

Google directly released a side-by-side comparison of Genie 3 and Genie 4.

On the left, Genie 3 is still a silent paper-thin figure with fused facial features and stiff limbs; on the right, Genie 4, anchored by Gemini Omni's real-world common sense, can already capture the warm light of a vanity mirror, an actor's micro-expressions, and even real-time reflections in the mirror, so delicate it looks like live-action long-take footage.

Wharton professor Ethan Mollick, who received early testing access, posted first-hand recorded gameplay.

He fed it just one sentence: "A neo-Gothic Brutalist city inhabited jointly by horses and robots," and in less than a minute, the city appeared.

Single-frame latency held steady at 23.4 to 25.3 FPS. With a casual click of the "dragon swoops across the sky" button, a dragon flaps its wings within seconds and roars past!

Still not satisfied, he then typed: "A friendly otter NPC walks over and asks a question."

Right after that, an otter wearing a vest really runs out from the street corner and speaks in a London accent: "Excuse me, traveler."

Then he posted another god-tier test: running DOOM inside Genie 4!

He first set the scene as "walking into an office where every computer and wall is running DOOM," then fired off three commands in a row: "A demon colleague appears," "He's carrying a Gatling gun," and "He gives me a bonus."

As soon as the words landed, a demon colleague really walked out, spinning a heavy Gatling gun in both hands, and handed out the bonus on the spot!

a16z partner Justine Moore also posted an underwater stealth test, saying it was a huge leap.

In an undersea cave, she clicked "cursed purple diamond glows" and "a swarm of electric eels attacks," and deep in the cave purple light instantly flashed wildly and a huge swarm of electric eels appeared.

One more click on "ghost pirate appears," and the undead pirate captain immediately showed up—so realistic.

Former Google AR/3D product manager Bilawal Sidhu, meanwhile, treated Genie 4 as a real-time virtual film set for a sci-fi blockbuster:

He summoned a UFO and typed, "It sucks me in"—the UFO instantly fired up a tractor beam, preparing to pull the person inside.

We can use natural language to modify the simulated scene in real time—for example, changing the weather, moving the camera, or introducing new characters or objects—and Gemini will adjust our instructions according to the surrounding scene. Between input commands, Gemini also monitors and guides the scene to ensure the whole world develops smoothly.

Sounds like Inception has come true?

A man plays the saxophone on an island; he types "whale tail appears," and a whale immediately appears in the water behind him.

Then, we make this world start snowing, and large goose-feather snowflakes immediately drift down.

If we pull the camera to a first-person perspective, the impact of this real-time rewriting of the world is amplified even further.

In this scene, we ride a giant dragon diving at 140 kilometers per hour, skimming past an old castle. If we command "the castle fires a fireball," the tower spews flames on the spot, and the fireball whizzes past us—the thrill is maxed out.

In the scene below, Genie 4 first generates the frozen Antarctic landscape based on the user's prompt, and then generates a first-person hot-air balloon scene.

Expanding the realistic boundaries of physics and human behavior

This time, Genie 4 introduces a brand-new standard of realism for interactive simulation.

Based on a super-strong understanding of world physics, Genie 4 simulates real-world physical scenes very realistically, such as aerodynamics, collisions, and so on.

And Genie 4's most notable advance lies in simulating people: it can create characters that move naturally, react to the environment, and can be interactively controlled in real time.

At a 1980s Japanese rural train station, give it a prompt, and the conductor will call out the departure announcement.

Give another prompt, and an old woman with a suitcase immediately appears.

The next scene makes us marvel at the extreme realism of Genie 4's physical interactions.

On a table in an outdoor courtyard, have a marble bust smash into a watermelon.

With a muffled thud, the flesh and juice splatter everywhere—from the gravitational impact to the trajectories of the fragments, Genie 4 reproduces it just right in real-time interaction.

Genie 4 also has a firm grasp of soft-body mechanics and aerodynamics.

Pink silk on a meadow—just blow a gust of wind, and it immediately billows into layer upon layer of silky waves in the wind, very realistic.

Even a close-up shot of a maple seed falling to the ground is incredibly realistic.

Push the camera to a macro perspective down to the millimeter, and the maple samara spins and glides down like a helicopter propeller, gently coming to rest among the blades of grass.

Farewell to the "silent wax museum"—Genie 4 can simulate sound!

Genie 4 is Google's first real-time world model that natively generates audio.

Each world has the sounds of its own environment, such as rustling leaves and waves lapping against the beach.

However, the real challenge is still human language! Genie 4 introduces real-time interactive dialogue. Characters engage in real-time conversation and react based on your input or events.

In this translucent 3D stained-glass world, a deer is summoned with a prompt. What's most striking is that its footsteps automatically turn into crisp sounds of colliding glass.

Hardcore architecture breakdown: three Geminis take turns, building a "hybrid world model" loop

If you simply treat Genie 4 as a "bigger video generator" with better image quality and sound, then you underestimate Google DeepMind's ambition.

Past real-time world models tended to collapse after just a few dozen seconds because pure pixel prediction lacks long-range logical planning—often after completing the first step, they forget all about the later goals.

To cure this chronic problem, Google has rolled out a highly disruptive "hybrid world model" architecture, sending three Gemini models to team up with Genie 4.

This adds a whole new reasoning layer to world simulation, enabling Genie 4 to simulate complex multi-step events and, before bringing AI into the real world, test how they would run in different environments.

The first layer of coordination is having Gemini 3.8 Flash act as the real-time "director" controlling the loop.

While Genie 4 generates frames one by one up front, Gemini 3.8 Flash watches the frames in the background in real time, breaking down the multi-step reasoning sequence, verifying whether the current step is complete while issuing the next instruction.

As the user intervenes, Gemini immediately adjusts, generating a world with reliable causality, programmer behavior, and multi-step reasoning.

Look at this close-up magic trick: each time the player closes the box door and waves the magic wand, Gemini 3.8 Flash in the upper right corner confirms the step is complete and directs Genie 4 to first turn the peacock into an orange cat, and then turn the orange cat into a retro radio!

Even if the script suddenly changes midway, this "director" can rearrange the script on the spot.

For example, in a world of ice and snow, have a meteor streak across, then direct a gray crowned crane to walk up to an electronic keyboard and peck out a melody, and finally have it soar into the sky.

If Gemini 3.8 Flash solves "how the world evolves logically," then the addition of Gemini 4 Argon and Gemini Robotics breaks through the ceiling of embodied intelligence and robot training.

When a human issues the command "play checkers with the boy across from you, move a red piece," Gemini 4 Argon on the left instantly breaks the task down into a 22-step, millimeter-level action script.

These high-density instructions are injected into Genie 4's generation stream in real time, driving the Apollo robot on the right to smoothly complete pinching the piece, making the move, and playing the game.

Genie 4 can even go into the lab to do precision scientific research.

Even an industrial-grade UR5 six-axis robotic arm can precisely grip a pipette in Genie 4 and insert it into a microplate to drip liquid.

Even if the target is moved away midway, the model can recalculate the strategy in real time and complete the grasp at the new position.

The necessary path to ASI is to first build a world

What is a true "world model"?

It's when you push open a door, turn around and leave for 30 seconds, then walk back, and the door is still open.

It can remember the laws of physics and maintain spatial memory—this is the watershed between video models and world models!

Now, Google Genie 4 has achieved it.

Relying on "massive unlabeled video pretraining + a small amount of action-trajectory fine-tuning," plus Gemini's symbolic logic and long-range memory, Genie 4 has pushed the duration of a single coherent world to about 2 minutes in one stroke.

From now on, building interactive environments completely bids farewell to cumbersome manual 3D modeling.

Genie 4 is now open to Google AI Ultra users, and will also open its doors to developers as the first world model ever to enter the Gemini API and Google AI Studio.

We can see Google's ambition: a true AGI can never live only inside a chat box!

References:

This article is from the WeChat public account "Xinzhiyuan" (ID: AI_era), author: Moses Aeneas, published by 36Kr with authorization.