Research Project

MoHuCo: Modeling Humans in Context

Type

National Project

Start Date

01/09/2021

End Date

31/08/2024

Project Code

PID2020-120049RB-I00

Project illustration

Staff

Project Description

Project PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033

Recent advances in computer vision and deep learning have shown impressive results in modelling different aspects of humans. Given a single image or a video sequence, these models provide detailed reconstructions of the body shape and clothes, predict future movements and understand human behaviour, emotions and intentions. However, one essential factor that has been obviated so far, is the fact that most of these human characteristics are inherently driven by interactions with objects and/or other people in the environment. For instance, the body trajectory is highly constrained by the spatial distribution of the rest of objects in the environment; a particular facial expression (e.g. ‘fear’) may respond to a specific circumstance occurring in the surrounding (e.g. ‘danger’). Understanding these types of human-context connections would allow going beyond current state-of-the-art and perform robust human reasoning under complex situations such as partial observations (e.g. crowded scenes, heavy occlusions) or indirect observations (predicting human characteristics from contextual clues).

The goal of MoHuCo is therefore to develop novel computer vision tools to discover interrelations between person’s properties and the context. For this purpose, we will split the project in three main blocks:

1) Observing the human: consolidation and pushing to the limits the algorithms for 3D body/cloth reconstruction, motion prediction and behaviour analysis given direct observations of the person.

2) Observing the context: research on novel algorithms to extract heterogeneous information (both geometric and semantic) of the environment; 3) Build joint human-context models: bringing the representations of humans and environment into a single model, allowing to indirectly reason about the human from direct observations of the context.

Project Publications

Conference Publications

  • A. Pérez and A. Agudo. Matching and recovering 3D people from multiple views, 2022 IEEE Winter Conference on Applications of Computer Vision, 2022, Waikoloa, Hawaii, USA, pp. 1184-1193, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo. Safari from visual signals: Recovering volumetric 3D shapes, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, Singapore, pp. 2495-2499.

    Open/Close abstract Abstract Info Info pdf PDF
  • D.F. Ordoñez, A. Agudo, F. Moreno-Noguer and M. Martin. An adaptable approach to learn realistic legged locomotion without examples, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 4671-4678.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ruiz, A. Agudo and F. Moreno-Noguer. Generating attribution maps with disentangled masked backpropagation, 2021 International Conference on Computer Vision, 2021, (Virtual), pp. 885-894.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Sanchez, A. Pumarola and F. Moreno-Noguer. PhysXNet: A customizable approach for learning cloth dynamics on dressed people, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 879-888.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Ramon, G. Triginer, J. Escur, A. Pumarola, J. García, X. Giro-i-Nieto and F. Moreno-Noguer. H3D-Net: Few-shot high-fidelity 3D head reconstruction, 2021 International Conference on Computer Vision, 2021, (Virtual), pp. 5620-5629, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Hernandez Ruiz, A. Vilalta and F. Moreno-Noguer. Neural Cellular Automata manifold, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10015-10023, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF
  • N. Ugrinovic, A. Ruiz, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Body size and depth disambiguation in multi-person reconstruction from single images, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 53-63.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Shen, A. Ruiz, A. Agudo and F. Moreno-Noguer. Stochastic Neural Radiance Fields: Quantifying uncertainty in implicit 3D representations, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 972-981.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Chatziagapi, S. Athar, F. Moreno-Noguer and D. Samaras. SIDER: Single-image neural optimization for facial geometric detail recovery, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 815-824.

    Open/Close abstract Abstract Info Info pdf PDF