World Labs releases Frameboy research preview: Atlas world model generates in real time at up to 15 fps from a single image

Click the generate button, then sit and wait for several minutes while the spinning progress bar does its thing. Anyone who has played with 3D world models or video generation in the past has probably been worn down into resignation by that kind of lengthy rendering.

Just today, World Labs, founded by Fei-Fei Li, released a research preview Demo called Frameboy, which directly pulls its own autoregressive world model Atlas into the era of real-time interaction.

All you need to do is open a browser, and cloud GPUs can stream the visuals to you in real time at frame rates of up to 15 fps, letting you freely control the camera like in a video game and wander at will through a 3D world generated from a single image.

Even if you accidentally head in the wrong direction or try a bad idea, the cost of trial and error drops to almost zero—one flick of the mouse and you can switch to another angle and keep exploring.

What's more interesting is that this update is also another technical answer delivered by Fei-Fei Li's top-tier spatial intelligence team after AMD announced at the end of September this year that it would acquire World Labs in an all-stock deal worth 8.2 billion US dollars (about 55 billion RMB).

Putting the world model into a retro handheld, running a single image in real time at 15 frames

To understand this update clearly, we first have to mention Atlas, which just debuted two months ago.

As an autoregressive world model, Atlas's biggest selling point at the time was that it could generate frames one by one and support pixel-level precision camera viewpoint control—whether it was imagining a virtual 3D world from a single image, or reconstructing real landmarks from several multi-view photos, checking progress at a construction site, or scouting locations for a film, it was all within reach.

But the problem was also very real: it was too slow.

If every adjustment of the camera position meant waiting tens of seconds or even several minutes to see the rendered result, creators would become constrained when building scenes and planning camera moves, and every press of the generate button would feel like opening a blind box.

To alleviate the long generation wait of world models, the team carried out deep inference optimization on Atlas and launched Frameboy, a real-time interactive Demo equipped with this optimized model; its retro handheld design also easily brings the classic Game Boy to mind.

Now, with image generation speed raised to up to 15 frames per second, users can clearly no longer feel that disjointed action of clicking generate.

You only need to push, pull, pan, and tilt the camera in the browser, and the cloud will continuously stream the visuals of new viewpoints to the screen in real time.

Take the official living room comparison demo: previously, when manually iterating with the slow model, users had to carefully click four keyframe camera positions in the room, press generate and wait for the video to come out, discover the angle was wrong, then drag and retry, going back and forth for several rounds.

In Frameboy's real-time mode, you just drive the virtual camera around the room, and wherever your field of view reaches, the visuals update synchronously in real time.

If you are familiar with World Labs' research preview project RTFM from last year, you might find the retro handheld form somewhat familiar.

However, compared with that lightweight frame generation model from back then, today's Atlas is not only far larger in model scale and has swallowed orders of magnitude more data, but its capability boundaries are also completely different.

Last year's RTFM could mainly only interpolate transitions between a few existing viewpoints, whereas today's Atlas, even if fed only a single static image or a text description, can imagine the entire surrounding scene from scratch and put it into Frameboy's spatial context for you to interact with and explore.

At the same time, it also fully retains and upgrades multi-view novel view synthesis: input several multi-angle photos of real-world famous landmarks and historic sites, and you can directly start a cyber real-scene roam in the browser.

Upload an image and build a 3D playground with your own hands

Just wandering through the few officially preset scenes is obviously not satisfying enough.

To let everyone build their own 3D worlds by hand, World Labs has also put real-time streaming capability into its web app Viewfinder, making it a real-time rendering overlay mode, which is currently in closed beta and has opened a waitlist application.

In Viewfinder, you can use an image you uploaded as a starting point, control the virtual camera to step outward bit by bit, and build the entire 3D space frame by frame.

Each time Atlas generates a new frame, it also outputs color and depth information synchronously, helping you visually align the entire three-dimensional spatial structure.

Once the main body of the scene is more or less built, you can directly summon the Frameboy overlay with one click, and the static project that was just being constructed frame by frame instantly turns into a real-time streaming 3D playground.

The team's positioning for the future is very clear: it wants world models like Atlas to become the next generation of generative renderers.

However, at the end of the blog post, the official team did not avoid the current computing threshold.

At this stage, Frameboy must run on high-performance GPU servers in the cloud, and compared with traditional local rendering, the cost is definitely not cheap.

But looking back at the history of computer graphics, any disruptive technology is often slow and expensive when it is first born; as underlying hardware and software algorithms evolve together, today's expensive cloud lab toy will sooner or later become an affordable tool for the masses.

A 55 billion marriage with Lisa Su, accelerating closer to underlying hardware

It is precisely on this matter of software-hardware collaboration to reduce cost and increase speed that today's World Labs can be said to have found the most suitable backer.

Just over a month ago, on September 28, chip giant AMD officially announced a major acquisition agreement, planning to take World Labs, founded only two years ago, into its fold in an all-stock deal worth 8.2 billion US dollars.

Expected to close by the end of this year, the deal directly set AMD's second-largest acquisition record in its history, after Xilinx.

By reckoning, this star unicorn headquartered in San Francisco and co-founded by Fei-Fei Li along with Justin Johnson, Ben Mildenhall, and Christoph Lassner was valued at 5 billion US dollars in its previous funding round.

According to the acquisition arrangement, after the deal is completed, Fei-Fei Li will officially serve as AMD executive vice president and chief scientist, reporting directly to Lisa Su; the original R&D team will be fully retained and continue to be led by Justin Johnson and Ben Mildenhall in grinding away at spatial models and physical AI.

At the time, outsiders were still discussing why a major hardware company that sells chips would spend 55 billion RMB to buy an algorithm team researching 3D world models.

Now, combined with Frameboy's thirst for extreme inference speed, the considerations on both sides are all too clear.

By taking in World Labs, AMD can immediately understand the real computing load characteristics of next-generation world models, thereby reverse-guiding the roadmap planning for its own GPU architecture, AI chips, and ROCm software stack, and going head-to-head with Nvidia in the robotics and spatial computing tracks.

In the past, running a real-time world model in the cloud meant wincing at expensive GPU bills; now, directly polishing hardware and low-level operators side by side with its own chip team, it is clearly another big step closer to truly putting real-time generative 3D rendering into terminal devices in millions of households.

This article comes from the WeChat public account "APPSO", author: APPSO, which discovers tomorrow's products, published by 36Kr with authorization.