Research Project
MoHuCo: Modeling Humans in Context
Type
National Project
Start Date
01/09/2021
End Date
31/08/2024
Project Code
PID2020-120049RB-I00
Staff
-
-
Agudo, Antonio
Principal Investigator
-
Sanchez, Jordi
Researcher
-
Pérez, Raül
PhD Student
-
Pérez, Marc
PhD Student
-
Gutiérrez, Marc
PhD Student
Project Description
Project PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033
Recent advances in computer vision and deep learning have shown impressive results in modelling different aspects of humans. Given a single image or a video sequence, these models provide detailed reconstructions of the body shape and clothes, predict future movements and understand human behaviour, emotions and intentions. However, one essential factor that has been obviated so far, is the fact that most of these human characteristics are inherently driven by interactions with objects and/or other people in the environment. For instance, the body trajectory is highly constrained by the spatial distribution of the rest of objects in the environment; a particular facial expression (e.g. ‘fear’) may respond to a specific circumstance occurring in the surrounding (e.g. ‘danger’). Understanding these types of human-context connections would allow going beyond current state-of-the-art and perform robust human reasoning under complex situations such as partial observations (e.g. crowded scenes, heavy occlusions) or indirect observations (predicting human characteristics from contextual clues).
The goal of MoHuCo is therefore to develop novel computer vision tools to discover interrelations between person’s properties and the context. For this purpose, we will split the project in three main blocks:
1) Observing the human: consolidation and pushing to the limits the algorithms for 3D body/cloth reconstruction, motion prediction and behaviour analysis given direct observations of the person.
2) Observing the context: research on novel algorithms to extract heterogeneous information (both geometric and semantic) of the environment;
3) Build joint human-context models: bringing the representations of humans and environment into a single model, allowing to indirectly reason about the human from direct observations of the context.
Project Publications
Journal Publications
-
E. Corona, G. Alenyà, G. Pons-Moll and F. Moreno-Noguer. LayerNet: high-resolution semantic 3D reconstruction of clothed people. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2): 1257-1272, 2024.
Abstract Info PDF
-
A. Dhamanaskar, M. Dimiccoli, E. Corona, A. Pumarola and F. Moreno-Noguer. Enhancing egocentric 3D pose estimation with third person views . Pattern Recognition, 138(109358), 2023.
Abstract Info PDF
Conference Publications
-
A. Agudo. Detail-aware uncalibrated photometric stereo, 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, Rhodes Island, Greece, pp. 1-5.
Abstract Info PDF
-
D.F. Ordoñez, M. Martin, A. Agudo and F. Moreno-Noguer. On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis, 2023 Robotics: Science and Systems Conference, 2023, Daegu, Republic of Korea.
Abstract Info PDF
-
A. Urdapilleta and A. Agudo. Comparative study of feature localization methods for endoscopy image matching, 2023 IEEE International Conference on Image Processing Challenges and Workshops, 2023, Kuala Lumpur, Malaysia, pp. 3719-3723, IEEE.
Abstract Info PDF
-
R. Pérez, A. Espersen and A. Agudo. Robust wind turbine blade segmentation from RGB images in the wild, 2023 IEEE International Conference on Image Processing, 2023, Kuala Lumpur, Malaysia, pp. 1025-1029.
Abstract Info PDF
-
M. Pérez and A. Agudo. Sensor-agnostic multimodal fusion for multiple object tracking from camera, radar, lidar and V2X, 2023 FISITA 2023 World Congress, 2023, Barcelona, to appear.
Abstract Info PDF
-
M. Pérez and A. Agudo. Robust multimodal and multi-object tracking for autonomous driving applications, 2023 International Conference on Advanced Robotics, 2023, Abu Dhabi, UAE, pp. 100-106, IEEE.
Abstract Info PDF
-
G.D. Delmas, P. Weinzaepfel, F. Moreno-Noguer and G. Rogez. PoseFix: correcting 3D human poses with natural language, 2023 International Conference on Computer Vision, 2023, Paris, France, pp. 14972-14982.
Abstract Info PDF
-
D.F. Ordoñez, M. Martin, A. Agudo and F. Moreno-Noguer. Morphological symmetries in robot learning, 2023 RSS Workshop on Symmetries in Robot Learning, 2023, Daegu (South Korea), pp. 1-5.
Abstract Info PDF
-
P. Caselles, E. Ramon, J. Garcia, X. Giro-i-Nieto, F. Moreno-Noguer and G. Triginer. SIRA: Relightable Avatars from a Single Image, 2023 IEEE Winter Conference on Applications of Computer Vision, 2023, Waikoloa, Hawaii, pp. 775-784.
Abstract Info PDF
-
P. Estevez and A. Agudo. Uncalibrated, unified and unsupervised specular-aware photometric stereo, 2022 ICPR Workshop on Towards a Complete Analysis of People: From Face and Body to Clothes, 2022, Montreal (Canada), pp. 7-20.
Abstract Info PDF
-
G.D. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer and G. Rogez. PoseScript: 3D human poses from natural language, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 346-362, 2022.
Abstract Info PDF
-
E. Corona, G. Pons-Moll, G. Alenyà and F. Moreno-Noguer. Learned Vertex Descent: a new direction for 3D human model fitting, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 146--165, 2022.
Abstract Info PDF
-
J. Shen, A. Agudo, F. Moreno-Noguer and A. Ruiz. Conditional-Flow NeRF: Accurate 3D modelling with reliable uncertainty quantification, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 540-557, 2022.
Abstract Info PDF
-
A. Pérez and A. Agudo. Matching and recovering 3D people from multiple views, 2022 IEEE Winter Conference on Applications of Computer Vision, 2022, Waikoloa, Hawaii, USA, pp. 1184-1193, IEEE.
Abstract Info PDF
-
A. Agudo. Safari from visual signals: Recovering volumetric 3D shapes, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, Singapore, pp. 2495-2499.
Abstract Info PDF
-
D.F. Ordoñez, A. Agudo, F. Moreno-Noguer and M. Martin. An adaptable approach to learn realistic legged locomotion without examples, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 4671-4678.
Abstract Info PDF
-
A. Agudo. Spline human motion recovery, 2022 IEEE International Conference on Image Processing, 2022, Bordeaux, France, pp. 4138-4142, IEEE.
Abstract Info PDF
-
N. Ugrinovic, A. Pumarola, A. Sanfeliu and F. Moreno-Noguer. Single-view 3d body and cloth reconstruction under complex poses, 17th International Conference on Computer Vision Theory and Applications, 2022, Online.
Abstract Info PDF
-
A. Ruiz, A. Agudo and F. Moreno-Noguer. Generating attribution maps with disentangled masked backpropagation, 2021 International Conference on Computer Vision, 2021, Montreal, Canada, pp. 885-894.
Abstract Info PDF
-
J. Sanchez, A. Pumarola and F. Moreno-Noguer. PhysXNet: A customizable approach for learning cloth dynamics on dressed people, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 879-888.
Abstract Info PDF
-
E. Ramon, G. Triginer, J. Escur, A. Pumarola, J. García, X. Giro-i-Nieto and F. Moreno-Noguer. H3D-Net: Few-shot high-fidelity 3D head reconstruction, 2021 International Conference on Computer Vision, 2021, Montreal, Canada, pp. 5600-5609.
Abstract Info PDF
-
A. Hernandez Ruiz, A. Vilalta and F. Moreno-Noguer. Neural Cellular Automata manifold, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10015-10023, Computer Vision Foundation.
Abstract Info PDF
-
N. Ugrinovic, A. Ruiz, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Body size and depth disambiguation in multi-person reconstruction from single images, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 53-63.
Abstract Info PDF
-
J. Shen, A. Ruiz, A. Agudo and F. Moreno-Noguer. Stochastic Neural Radiance Fields: Quantifying uncertainty in implicit 3D representations, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 972-981.
Abstract Info PDF
-
A. Chatziagapi, S. Athar, F. Moreno-Noguer and D. Samaras. SIDER: Single-image neural optimization for facial geometric detail recovery, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 815-824.
Abstract Info PDF
Follow us!