Import AI 479: Realtime-Bench reveals LLM shortcomings on real-time tasks, DIA space security report, and…

Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4

What do LLMs think about vision models?Jack ClarkNov 02, 2026

LLMs are very bad at low-latency tasks: …Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you… Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation).

Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks.

What it measures: The test is designed to probe for three qualities: temporal state awareness (”RT-Perceive”), where models can use observations to track changes in the world; adaptive deliberation (”RT-Deliberate”), where models alter their reasoning length and depth according to available time; and time-aware action selection (”RT-Act”) where models need to appropriately time their actions for the situation. “For example, an agent purchasing a ticket must track the sale status and assess whether observed availability is still current (RT-Perceive), limit further deliberation when the purchase window is closing (RT-Deliberate), and choose between waiting for sales to open and purchasing while tickets remain available (RT-Act).” They have three different conditions: Wall clock, which is designed to test out performance under real-world latency. Paused clock, which lets you arbitrarily pause things so you can get a sense of baseline capability, and Token clock, which “advances the environment at a fixed rate per generation token. An action is applied at the simulated time reached when it is committed, so consuming more tokens delays execution regardless of actual inference latency”.

How well do models do? The authors tested out models from OpenAI (GPT-6-Astra, GPT-5.6 Sol, GPT-Realtime-2, OpenAI Decisions), Anthropic (Claude Fable 5.1, Claude Opus 5), Google (Gemini 3.8 Flash, Gemini 3.1 Flash Live, Gemini 3.8 Live), Jev 1.13, as well as Chinese models from Deepseek (V4.1) and Moonshot AI (Kimi-K3). GPT-6 Astra (high) got the highest pass rate on the ‘wall clock’ evaluation setting, with 36%. By comparison, with a paused clock, models with high reasoning get closer to 70% or even 80% success.

Why this matters: This benchmark is an effective proxy for where AI agents will break - the low scores here point to tasks where AI agents cannot yet be expected to perform well. Check out the leaderboard: Realtime-Bench (official website). Read the paper: Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems? (arXiv).

***

The future of spacewar, courtesy of the Defense Intelligence Agency: …Anti-satellite tech, space nukes, and more… Working in AI can be stressful so sometimes to relax myself I like to read about other topics to get a sense for what horrors lie elsewhere. I take the same approach with depression, where if I feel sad I will try and listen to music by people sadder than me in an attempt to cheer myself up (e.g., Radiohead, Neil Young, and in very dark moments Bill Callahan / Smog, etc). So with that in mind I recently read “Challenges to Security in Space”, by the Defense Intelligence Agency.

What’s going on in space: As with conflict on Earth, space is defined by conflict between the US versus China, Russia, Iran, and North Korea, with China the most technologically sophisticated, and Russia the most seemingly gung ho. “China and Russia are developing a formidable spectrum of offensive capabilities to degrade or destroy space services of other countries, from kinetic destruction to sophisticated jamming and cyber operations,” the DIA writes. “China has more than tripled its space launches per year in the past decade, with 22 launches in 2016 and 93 launches in 2025… Beijing judges space is central to the United States and its allies’ operations. Beijing also considers space an offensive military domain, which has shaped integral components of PLA military planning and campaigns.”

Spacewar: The general vibe you get from reading this 100+ page report is there is a huge amount of ugly, frightening stuff happening in space, or being done to things in space by things on the ground. Some example quotes:

“By the late 2020s, China may field space-based directed energy systems and higher-powered ground-based directed energy systems capable of threatening satellites in higher orbits”

“China is developing space-based technology, including on-orbit counterspace weapons, capable of holding space systems at risk”

“Russia’s Peresvet, a ground-based laser system, is designed to dazzle enemy satellites’ optical systems to mask the movement of Russia’s strategic missile systems”

“Russia has launched several satellites over the last few years which conducted test activities on orbit for technologies that could enable attacks on other countries’ satellites and result in temporary or permanent damage”

China has a couple of spaceplanes, akin to the US’s hyper-secret X-37B vehicle, and they appear to be getting more sophisticated. In 2023, one of China’s aforementioned vehicles, “released and interacted with an object.”

Satellite-on-satellite fights: It’s obvious throughout the report that we’re in a world where rival nations have satellites that try to go up and mess with other satellites from other people.

“In 2019, Russia launched satellite Cosmos 2542, which released a second satellite, Cosmos 2543. Cosmos 2543 then began following a U.S. national security satellite and approached it multiple times at a distance close enough to create potentially dangerous operating conditions,” the DIA writes. “Russia is developing kinetic orbital systems to permanently disable, or “kill,” satellites. Moscow’s Burevestnik is a developmental, co-orbital air-launched ASAT weapon for killing spacecraft in LEO”.

“In February 2022, Russia launched a satellite called Cosmos, which Moscow claimed at the time was testing the effects of space radiation on electronic systems. This satellite was launched into an unusual and isolated orbit not used by any other spacecraft”.

Space nukes: Russia is experimenting with nukes in space. “A nuclear detonation in space would create an electromagnetic pulse, which would immediately affect satellites within range and could impact civilian infrastructure,” the DIA writes. “A nuclear detonation in space could be tailored to impact satellites in most orbits—LEO for commercial and reconnaissance satellites, GEO for critical communications, and MEO for navigation and timing satellites”.

***

Metis: A startup trying to fuse symbolic and neural reasoning: …Towards Vulcan-like machine minds… Metis, a startup that wants to build novel AI architectures oriented around developing machines with better reasoning, has published a research paper laying out its research approach. The high-level summary is simple - Metis wants to do for modern open-domain problems what DeepMind did for closed-domain problems like board games via its work on AlphaGo and AlphaGo Zero.

What they want to do: The key idea here is to add a reasoning core to modern AI architectures which is far more reliable than what we have today. “In modern AI architectures much of the other surrounding machinery that made AlphaGo a successful reasoning system remains either implicit or absent,” they write. “The first missing element is an explicit and persistent epistemic state… the second missing element is a criterion that allows the system to choose reasoning moves that make progress towards the goal.” This is more of a position paper than a technical paper, so they don’t have a ton of detail here. The main idea is to train a reasoning policy and value function based on real-world data collection, then use this to guide the system to perform reasoning that reduces long-term uncertainty, and have these sit on a basic core of data made up of claims or hypotheses about the world. “Any change to a claim’s evidential status must be licensed by evidence or verified through explicit calculation and approved by an independent judgment system,” they write. “Model-generated content cannot enter the accepted epistemic state merely by being plausible; it must pass warrant assessment, which lowers hallucinations. Sources of evidence can be attached directly to the propositions they support, providing a clear chain of provenance. The reasoning process becomes explicitly auditable.”

***

Key features: Google calls out four features that make Genie 4 different to previous ones. These include:

Real-time modification: “You can use natural language to modify a simulation in real-time — like changing the weather, moving the camera, or introducing a new person or object — and Gemini will contextually adapt your prompt to match the surrounding scene.”

Physics and human action simulation: “Genie 4 simulates real-world physics scenarios, such as aerodynamics, collisions, and more,” and it also simulates people better. “The most dramatic leap is how Genie 4 simulates people, creating characters that move naturally, respond organically to their environments, and can be interactively steered in real-time.”

Sound modeling: “Genie 4 is our first real-time world model to generate audio natively,” Google writes. It also “introduces interactive dialogue that adapts on the fly.”

Simulating complex events: Google has embedded Gemini 3.8 Flash into Genie 4, so that the fast Flash model “serves as a guiding agent in the loop”, providing continual calibration. “Gemini continuously observes the scene, breaks down goals into a sequence of steps, and reviews incoming frames to track the progress and verify that each step succeeded. As the world shifts or the user intervenes, Gemini adapts its plan in the moment, unlocking worlds with reliable causality, procedural behaviors, and multi-step reasoning.”

Why this matters - towards a multitude of worlds all dreamed up by machines: Though the technology is currently quite janky (e.g., robotic voices, a dreamlike sort of inconsistency in the visualized environments, etc), it points towards a world where we can generate infinite forms of entertainment (and potentially data?) via creating and prompting custom worlds. Read the announcement post: Genie 4: A new generation of world simulation (Google DeepMind blog). Watch the video: Genie 4: A new generation of world simulation (Google DeepMind, YouTube).

Tech Tales:

City Race [Description of a game played in the 2040s] One of the favorite games that children would play after The Uplift was called City Race; in it, they competed against one another to plan, build, and manage cities, expanding across the domain of a novel planet.

The cities were trapped in time, starting with the technology of whatever era they were in, and the games took place over the course of about a century, allowing some further technology development to occur.

Each game was played using a slice of the resources from the Dyson Swarm, allowing for a powerfully complex simulation (albeit with local-agent cognitive gating, such that nothing could develop enough sophistication to be deemed a Conscious Entity).

Hundreds of thousands of simulated minds lived and died, sometimes thriving in cities run by careful and effective owners, and other times trapped in squalor, or growing old in places whose economies were hollowing out; casualties of the successes of others.

It was deemed good for the children to learn what it meant to be gods, before they became adults. At the closure of every game, the children would need to spend a day living inside each of the cities in the gameworld they had played; those who had built marvelous cities experienced the wonders of their own creation, then walked through the broken towns whose success had been stymied by their own. Those who had struggled lived in their twilight towns, then got to tour the far off places that had doomed them.

Things that inspired this story: Player of Games by Iain M Banks; Simcity; game playing; urban design; The Glass Bead Game by Hermann Hesse.