Abstract
Creating scenes for captured motions that achieve realistic human-scene interaction is crucial for 3D animation in movies or video games. As character motion is often captured in a blue-screened studio without real furniture or objects in place, there may be a discrepancy between the planned motion and the captured one. This gives rise to the need for automatic scene layout generation to relieve the burdens of selecting and positioning furniture and objects. Previous approaches cannot avoid artifacts like penetration and floating due to the lack of physical constraints. Furthermore, some heavily rely on specific data to learn the contact affordances, restricting the generalization ability to different motions. In this work, we present a physics-based approach that simultaneously optimizes a scene layout generator and simulates a moving human in a physics simulator. To attain plausible and realistic interaction motions, our method explicitly introduces physical constraints. To automatically recover and generate the scene layout, we minimize the motion tracking errors to identify the objects that can afford interaction. We use reinforcement learning to perform a dual-optimization of both the character motion imitation controller and the scene layout generator. To facilitate the optimization, we reshape the tracking rewards and devise pose prior guidance obtained from our estimated pseudo-contact labels. We evaluate our method using motions from SAMP and PROX, and demonstrate physically plausible scene layout reconstruction compared with the previous kinematics-based method.
Method Overview
Given a set of interacting objects and a motion sequence, we aim to generate scene layouts with physically plausible object placement that aligns with human actions. At a high level, our proposed framework formulates scene layout generation as an optimization problem for maximizing the motion tracking score in physics-based simulation. The optimization process comprises two key components: learning a motion imitation controller to animate the simulated character within a physics simulator and updating the scene layout generator to provide the physical affordances for the interaction.
Physically Plausible Scene
Illustrations of our generated scenes with adapted SUMMON for comparison. Our method generates reasonable and plausible scenes.
Diverse Generation
We demonstrate the ability of INFERACT to generate diverse results for various motions.
Optimal Object Selection
We show the INFERACT is able to select the most proper object for different motions.
Complex Interaction
We demonstrate the INFERACT can be applied to more complicated interactions like outdoor vaulting action.
Citation
If you find this project helpful, please cite us: