LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field

Jiyang Li1*, Lechao Cheng2*, Zhangye Wang1, Tingting Mu3, Jingxuan He2✉
1Zhejiang University, 2Hefei University of Technology, 3The University of Manchester
*Equal contribution. ✉Corresponding author.
Paper arXiv Code Data

LoopGaussian convert multi-view images of a stationary scene (a) into authentic 3D cinemagraph by an Eulerian motion field (b). The 3D cinemagraph can be rendered from a novel viewpoint to obtain a natural seamless loopable video (c).

Abstract

Cinemagraph is a unique form of visual media that combines elements of still photography and subtle motion to create a captivating experience. However, the majority of videos generated by recent works lack depth information and are confined to the constraints of 2D image space. In this paper, inspired by significant progress in the field of novel view synthesis (NVS) achieved by 3D Gaussian Splatting (3D-GS), we propose \textbf{\textit{LoopGaussian}} to elevate cinemagraph from 2D image space to 3D space using 3D Gaussian modeling. To achieve this, we first employ the 3D-GS method to reconstruct 3D Gaussian point clouds from multi-view images of static scenes, incorporating shape regularization terms to prevent blurring or artifacts caused by object deformation. We then adopt an autoencoder tailored for 3D Gaussian to project it into feature space. To maintain the local continuity of the scene, we devise SuperGaussian for clustering based on the acquired features. By calculating the similarity between clusters and employing a two-stage estimation method, we derive an Eulerian motion field to describe velocities across the entire scene. The 3D Gaussian points then move within the estimated Eulerian motion field. Through bidirectional animation techniques, we ultimately generate a 3D Cinemagraph that exhibits natural and seamlessly loopable dynamics. Experiment results validate the effectiveness of our approach, demonstrating high-quality and visually appealing scene generation.

-->

Comparisons

LoopGaussian (below) vs. 3D Cinemagraph (above)

Novel View Synthesis

3D Cinemagraph generated by our method can be rendered from any viewpoint.

Optical Flow Maps

The rendering results and corresponding optical flow maps.

BibTeX

@article{li2024loopgaussian,
  title={LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field},
  author={Li, Jiyang and Cheng, Lechao and Wang, Zhangye and Mu, Tingting and He, Jingxuan},
  journal={arXiv preprint arXiv:2404.08966},
  year={2024}
}