KAIST and Google DeepMind open-source FL4N: real-time reconstruction of 4D dynamic scenes from monocular video
The Korea Advanced Institute of Science and Technology (KAIST) and Google DeepMind jointly proposed FL4N, a 4D novel view synthesis model that can reconstruct dynamic scenes from monocular video and render images from different viewpoints and time points in real time. The paper has been published on arXiv, and the code and model weights have been open-sourced simultaneously. FL4N adopts feed-forward latent space modeling, introducing a lightweight dynamic encoder based on a pretrained static scene model, and learns object motion through temporal residuals, reducing reliance on complex 3D motion annotations. On the NVIDIA Dynamic Scenes, N3DV, and Kubric4D datasets, it outperforms the compared 4D Gaussian methods on multiple image quality metrics, and is expected to be used in XR content production, immersive video, and dynamic digital twins.
Body · AI translation
Recently, researchers from the Korea Advanced Institute of Science and Technology (KAIST) and Google DeepMind jointly proposed FL4N, a 4D novel view synthesis model that can reconstruct dynamic scenes from monocular video and render images from different viewpoints and time points in real time. The related paper has been published on arXiv, and the project code and model weights have been open-sourced simultaneously.
It is understood that traditional 3D reconstruction mainly targets static scenes, whereas 4D reconstruction needs to further capture the motion changes of objects over time. Some existing methods rely on explicit representations such as 4D Gaussian, and when handling complex motion scenes, they are prone to issues such as ghosting, artifacts at object edges, and insufficient temporal consistency.
To address the above challenges, FL4N adopts a feed-forward latent space modeling approach, encoding the spatial structure and temporal changes of the scene into a unified latent representation. Based on a pretrained static scene model, the model introduces a lightweight dynamic encoder that learns object motion and scene changes through temporal residuals, reducing reliance on complex 3D motion annotations.
In terms of technical implementation, FL4N uses static scene features to assist dynamic modeling, distinguishes between the object's own motion and the parallax caused by camera movement, and supports independent adjustment of the viewing viewpoint and scene time, enabling continuous synthesis of dynamic imagery.
In experiments, the research team conducted tests on datasets including NVIDIA Dynamic Scenes, N3DV, and Kubric4D. The paper shows that FL4N outperforms the compared 4D Gaussian methods on multiple image quality metrics and possesses real-time novel view rendering capability, with potential applications in areas such as XR content production, immersive video, and dynamic digital twins.
Source: FL4N official project homepage
Submissions/Tips: tougao@youxituoluo.com
Manuscripts/Business cooperation: Lin Nan (WeChat 19250561593)
Join the industry exchange group: Lin Nan (WeChat 19250561593)
Metaverse Digital Industry Service Platform
Download the "Tuoluo Technology" APP to get cutting-edge in-depth metaverse information