Google DeepMind releases the world model Genie 4, supporting real-time interaction and native audio

Google's newly released Genie 4 is turning this vision into reality.

On October 27 local time, Google DeepMind officially launched its next-generation world model Genie 4, calling it "the most realistic and interactive world model to date."

Compared with the previous Genie 3, which mainly broke ground in real-time scene generation and interaction, the new model further introduces native audio generation, real-time natural language control, and simulation of complex physics and character behavior, and uses the Gemini series of models to achieve multi-step event reasoning.

In the official demo, users use the keyboard to control navigation and the camera, and use text prompts to change the scene: change the weather, move the camera, add people or objects to the scene. For example, make a pink silk ribbon appear on a grassy field, or place the camera inside a hot air balloon basket above the Antarctic ice sheet. The scene responds in real time, and the characters continue to act according to the new settings.

Behind this capability, the Gemini series of models takes over. Gemini Omni is responsible for making the visuals closely match the appearance and behavior of the real world, while other Gemini models are responsible for translating requests into actions, planning complex events, observing the visuals, and tracking progress. When users change instructions, Gemini adjusts the plan on the spot.

In another set of demos, Gemini 4 Argon even began directing Genie 4, step by step simulating the process of a humanoid robot playing checkers.

As can be seen, this is no longer just AI video generation in the traditional sense, but a move toward an interactive, controllable world simulation system that can be used for AI training. Currently, Genie 4 has opened a research preview to a small number of trusted testers, and will gradually be provided to Google AI Ultra subscribers, as well as early partners of the Gemini API and Google AI Studio.

In terms of parameters, a single generation of Genie 4 can provide an interactive experience of up to 2 minutes, while the upper limit for the previous generation Genie 3 was 1 minute. This number may look unremarkable, but for game developers validating a single segment of gameplay and film and television creators previewing a single set of shots, it is already enough to cover the most basic testing needs.

This time, Google demonstrated capability upgrades in four areas: real-time world changes, simulation of physics and character behavior, native sound generation, and complex event reasoning combined with Gemini.

Some of these demos are already quite close to a combination of interactive games and real imagery.

First, users do not need to design the entire scene in advance and can change the world at any time during the generation process.

In one demo, a saxophone player in a brown suit stands on a floating platform in the middle of a lake. Surrounding him are a misty forest, slowly flowing lake water, and a sphere emitting warm light.

The character keeps playing, with fingers, breathing, and body swaying naturally connected, while the camera can move independently to observe the entire environment from different positions. The key here is not making the visuals look like a film, but allowing the camera to freely explore the scene while maintaining the spatial relationships among the character, objects, and surrounding environment.

Users can also control movement and perspective through the keyboard, and use text instructions to change the weather in real time and introduce new characters or objects. Gemini understands these requests based on the current environment and helps maintain the development of the scene when the user stops providing input.

Second, objects are no longer just decorations in the frame; characters and the environment begin to exhibit more complex patterns of motion.

In another batch of cases shown by Google, a whole watermelon is placed on a stone platform, and maple seeds spin down into the grass; there is also a rural Japanese train station crowded with passengers, where people walk and wait on the platform, together with the surrounding environment forming a dynamic scene.

Google emphasizes that Genie 4 has improved its ability to simulate physical phenomena such as aerodynamics and collisions, with particularly notable progress in character movement. The model can generate more natural human motion, allowing characters to react to their surroundings and accept real-time control.

However, the physics simulation here is still the model generating results based on the world patterns it has learned, which does not mean that all physical processes are precisely calculated.

Third, Genie 4 gives an AI world natively generated sound for the first time.

In the demo video, you ride on a dragon's back through mountains, surrounded by other riders. You feel as if you are sitting on a giant dragon's back, crossing the mountains at the speed of 140 kilometers per hour set by the prompt, and the WASD prompt in the lower left corner tells you that you really are "playing" this generated scene.

Accompanying the flight are not only rapidly passing mountain peaks and castles, but also the roar of the giant dragon's wings, the howling airflow at high altitude, and the sound of volcanic activity in the distance. These sounds are not added later in post-production for the demo, but are generated directly by Genie 4.

Furthermore, the model can also generate character dialogue in real time. Characters in the scene can respond verbally according to user input or events occurring in the world, rather than merely repeating pre-prepared lines.

For virtual characters, interactive storytelling, and even game development, this change is quite important. What developers need to create may no longer necessarily be characters with every line and every reaction hard-coded, but rather simulated characters that can continuously produce behavior and dialogue as the situation changes.

In a video actually generated by developer Justine Moore (who is also a partner at a16z), a more complex, fully real-time generated game world is shown. The entire scene is generated from a single prompt—environment plus characters—and then events can be changed with prompts.

In the video, a character in a diving suit explores a cave, with gold coins scattered underfoot and a wooden barrel nearby. At the top of the screen there is prompt text reading "Describe changes to your world...", along with several task options such as Ai Suggested and Giant Kraken Reach. The frame rate panel on the right shows the current 24.6 FPS, a minimum of 17, and an average of 23.8.

Justine Moore said she had early access to Genie 4, and that creating a world usually takes less than 30 seconds, with a real-time interaction window of about 2 minutes. Users can move via keyboard and mouse, and can also change the world using natural language—for example, modifying the weather, moving the camera, or adding a character. Gemini-series models process these instructions to ensure the coherence of the scene.

In addition, Gemini4 Argon breaks tasks down into a series of steps, and then guides Genie 4 through text instructions to gradually present the execution process. As the simulation progresses, the system can continue to adjust subsequent steps. This is equivalent to letting a model skilled in reasoning and planning direct another model skilled in real-time environment generation.

Genie4 technology revealed

From a technical perspective, Google is trying to solve two long-standing difficulties faced by world models: how to make the simulated environment change in real time with user operations, and how to keep continuously occurring events logically consistent. Behind this, two brains play a role: Gemini is responsible for thinking, and Genie is responsible for generating the physical world.

To understand this change, one must first clarify the difference between world models and ordinary video models.

- Traditional video generation models mainly generate imagery over a relatively fixed time range based on conditions such as text and images.

- World models need to further answer: if the user walks to the left, what will happen to the field of view? If someone knocks over an object on the table, what happens next? If a character leaves the room and then comes back, does the previous environment still exist?

This capability involves two key concepts: Action-conditioned Generation and World Consistency.

The former means that the next frame cannot be predicted based only on the previous frame; it must also take into account what action the user has just taken. The latter requires the model to remember the environment it has already generated, and try as much as possible to avoid characters, objects, or spatial relationships changing for no reason over time.

In fact, when Google released Genie 3 in 2025, it had already achieved real-time interactive generation at 24 frames per second and 720p resolution, and supported environment simulation lasting several minutes. Genie 3 uses frame-by-frame autoregressive generation and does not rely on building a complete explicit 3D scene in advance; instead, it continuously generates subsequent content based on previous frames and user operations.

But this also brings new technical challenges: the longer the generation time, the more historical information the model needs to reference, and errors may also accumulate continuously. For example, if the user passed a certain building a minute ago and now walks back, the model must try as much as possible to keep the building's appearance and position unchanged.

Genie 4 continues the direction of real-time world generation, but the most prominent change this time is the introduction of Gemini-series models into the simulation process. According to official disclosures, the entire system involves capabilities at at least three different levels.

At the first level, Genie 4 is responsible for real-time generation of visual, audio, and dynamic environments. It needs to continuously generate new visuals and sounds based on the current world state and inputs, and represent character movement, environmental changes, and interaction results.

At the second level, Gemini Omni strengthens the connection between the simulated world and real-world laws. Google says that GeminiOmni provides the foundation of real-world appearance and behavior for the simulation.

At the third level—also the most important breakthrough this time—Gemini 3.8 Flash serves as a real-time directing agent. While Genie 4 continuously generates the environment, Gemini 3.8 Flash constantly observes the scene, breaks complex goals down into multiple steps, and then judges whether the current step has been completed by checking the newly generated frames.

Once the user intervenes or the world state changes, Gemini can readjust the plan. The importance of this mechanism lies in introducing the language reasoning capability of large models into the process of dynamic environment generation. In the past, even if a world model could generate realistic robot actions, it might not be able to ensure correct task logic among multiple actions. Now, Gemini can continuously provide planning, checking, and adjustment, giving the system a better chance to maintain the coherence of complex events.

It is worth noting that Google is also exploring further combinations.

In the demonstration of the Apollo humanoid robot playing chess, the one participating in task planning is Gemini 4 Argon. It splits the user's goal into steps, then hands the corresponding text instructions to Genie 4 to drive the simulation process.

Google calls this route the future of "Hybrid World Models." Its basic idea is to combine the structured logical reasoning that code and language models excel at with the rich sensory simulation capabilities that pixel-level generative models excel at.

In other words, the former is responsible for understanding how events should develop, and the latter is responsible for presenting what this process might look like in the real world.

This article comes from the WeChat public account "AI Frontline" (ID: ai-front), author: Qinghe, editor: Siyue, published by 36Kr with authorization.