VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

VR-GS empowers users to interactive manipulate 3D Gaussian kernels at a real-time rate with physically plausible response and realistic illumination.

Abstract

As consumer Virtual Reality (VR) and Mixed Reality (MR) technologies gain momentum, there's a growing focus on the development of engagements with 3D virtual content. Unfortunately, traditional techniques for content creation, editing, and interaction within these virtual spaces are fraught with difficulties. They tend to be not only engineering-intensive but also require extensive expertise, which adds to the frustration and inefficiency in virtual object manipulation. Our proposed VR-GS system represents a leap forward in human-centered 3D content interaction, offering a seamless and intuitive user experience. By developing a physical dynamics-aware interactive Gaussian Splatting in a Virtual Reality setting, and constructing a highly efficient two-level embedding strategy alongside deformable body simulations, VR-GS ensures real-time execution with highly realistic dynamic responses. The components of our Virtual Reality system are designed for high efficiency and effectiveness, starting from detailed scene reconstruction and object segmentation, advancing through multi-view image in-painting, and extending to interactive physics-based editing. The system also incorporates real-time deformation embedding and dynamic shadow casting, ensuring a comprehensive and engaging virtual experience.

Pipeline

VR-GS is an interactive system meticulously crafted to integrate 3D Gaussian Splatting (GS) and eXtended Position-based Dynamics(XPBD) for generating a real-time interactive experience in Virtual Reality (VR). Beginning with multi-view images, the pipeline skillfully combines scene reconstruction, segmentation, and inpainting using Gaussian kernels. These kernels form the foundation for VR-GS's utilization of the sparse volumetric data structure VDB, facilitating bounding mesh reconstruction and subsequent tetrahedralization. VR-GS further harnesses a novel two-level rendering geometry embedding, XPBD, collision detection, and shadow casting techniques, all converging to deliver a captivating and immersive user experience.

Generated dance in VR.

Toss a ring over a human.

Pet a fox.

VR-GS enables users to interact with 3D Gaussian kernels at a user-friendly manner.

Visual Quality Comparison: our method synthesizes competitive visual results compared to PhysGuassian and significantly outperforms PAC-NeRF.

BibTeX

@article{jiang2024vr-gs,
      title={VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality},
      author={Jiang, Ying and Yu, Chang and Xie, Tianyi and Li, Xuan and Feng, Yutao and Wang, Huamin and Li, Minchen and Lau, Henry and Gao, Feng and Yang, Yin and Jiang, Chenfanfu},
      journal={arXiv preprint arXiv:2401.16663},
      year={2024},
}