Publication

Robust spatio-temporal clustering and reconstruction of multiple deformable bodies

Journal Article (2019)

Journal

IEEE Transactions on Pattern Analysis and Machine Intelligence

Pages

971-984

Volume

41

Number

4

Doc link

https://doi.org/10.1109/TPAMI.2018.2823717

File

Download the digital copy of the doc pdf document

Abstract

In this paper we present an approach to reconstruct the 3D shape of multiple deforming objects from a collection of sparse, noisy and possibly incomplete 2D point tracks acquired by a single monocular camera. Additionally, the proposed solution estimates the camera motion and reasons about the spatial segmentation (i.e., identifies each of the deforming objects in every frame) and temporal clustering (i.e., splits the sequence into motion primitive actions). This advances competing work, which mainly tackled the problem for one single object and non-occluded tracks. In order to handle several objects at a time from partial observations, we model point trajectories as a union of spatial and temporal subspaces, and optimize the parameters of both modalities, the non-observed point tracks, the camera motion, and the time-varying 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and does not require any training data at all. We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction. We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal segmentation.

Categories

computer vision, optimisation.

Author keywords

Non-Rigid Structure from Motion, Union of Subspaces, Spatio-Temporal Clustering, Augmented Lagrange Multipliers

Scientific reference

A. Agudo and F. Moreno-Noguer. Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4): 971-984, 2019.