Research Project

HuMoUR: Markerless 3D human motion understanding for adaptive robot behavior

Type

National Project

Start Date

01/01/2018

End Date

30/06/2021

Project Code

TIN2017-90086-R

Default project illustration

Staff

Project Description

The goal of HuMoUR is to develop novel computer vision tools to estimate and understand human motion using a simple camera and use this information as a demonstration to teach a general purpose robotic assistant to perform new complex manipulation tasks. In Robotics, this learning paradigm is referred to as Learning from Demonstration: a non-expert teacher repetitively executes a task so the robot can learn the steps and variability of the actions. Typical setups take place in controlled laboratory facilities and consist of a manipulator arm teleoperated by the user through a haptic device. In order to bring this technology to the next stage of development and out of the laboratory, we believe it is paramount to contribute both on the sensing and action fronts of the problem. HuMoUR will advance both these fields.

Specifically, on the sensing side we will: (1) research novel markerless methodologies to capture 3D human pose and motion from monocular cameras. We will leverage on current Deep Learning (DL) strategies to make these algorithms view-point invariant and reliable on images acquired in-the-wild. One important aspect that will be investigated is that of integrating geometric priors within the DL formulations in order to simultaneously exploit physical models and statistical evidence of the data; (2) Additionally, we will explore the use of Convolutional and Recurrent Networks to design new motion prediction algorithms able to infer the future position of the human body. (3) And finally, pose and motion estimates will be exploited to devise new strategies for 3D human action recognition.

The outcomes of the sensing modules will be the primary key to endow service robots with new features and learning possibilities. On this regard, we aim to (4) adapt existing reinforcement learning algorithms such that they can be carried out at end-users home with demonstrations recorded by one single camera; (5) propose new planning strategies to account for robot adaptation to user requirements, and to contacts between the robot and the environment/people; and (6) implement new protocols that ensure human safety in the tasks where there is a close interaction with the robot. 3D human motion prediction algorithms will play an essential role to deploy such protocols.

We plan to demonstrate our developments on three main scenarios: (a) Feeding a person and (b) brushing a persons hair, where stable spoon/brush trajectories need to be adapted to the pose of the head. (c) Help dressing a person, where the perception algorithms will need to tackle strong body occlusions caused by clothes and the robot will need to rapidly react to sudden changes of the persons pose. The main objectives we pursue are commercially and socially relevant robotics technologies, as endorsed by our EPOs. In particular, the project responds the demand for new technologies for the assistance of elderly and disabled people, one of the main pillars of EU H2020 plan.

Project Publications

Journal Publications

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. How do consumers' gender and rational thinking affect the acceptance of entertainment social robots?. International Journal of Social Robotics, 14: 973–994, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Learning in autonomous and intelligent systems: Overview and biases from data sources. Arbor - Ciencia, Pensamiento y Cultura, 197(802): a627, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. The effects of gender and personality of robot assistants on customers’ acceptance of their service. Service Bussiness, 16: 359–389, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, V. Lepetit and F. Moreno-Noguer. Simultaneous completion and spatiotemporal grouping of corrupted motion tracks. The Visual Computer, 38(11): 3937–3952, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 519-532, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, H. Siqueira, D. Fu, S. Magg, P. Barros, S. Wermter, C. Torras and G. Alenyà. Do I have a personality? Endowing care robots with context-dependent personality traits. International Journal of Social Robotics, 13(8): 2081-2102, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, C. Torras and G. Alenyà. Are preferences useful for better assistance?: A physically assistive robotics user study. ACM Transactions on Human-Robot Interaction, 10(4): 1-19, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Short-term human-robot interaction adaptability in real-world environments. International Journal of Social Robotics, 12: 639-657, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128: 698-713, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Cognitive system framework for brain-training exercise based on human-robot interaction. Cognitive Computation, 12: 793-810, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, G. Alenyà and C. Torras. Adapting robot task planning to user preferences: An assistive shoe dressing example. Autonomous Robots, 43(6): 1343-1356, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4): 971-984, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, S. Foix and G. Alenyà. On inferring intentions in shared tasks for industrial collaborative robots. Electronics, 8(11): 1306, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Shape basis interpretation for monocular deformable 3-D reconstruction. IEEE Transactions on Multimedia, 21(4): 821-834, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Simao, F. Martínez-Jerónimo, V. Blasco, F. Moreno-Noguer, J.M. Porta, J. Pestana, A. Soarez, D. Raldúa and C. Barata. Using a new high-throughput video-tracking platform to assess behavioural changes in Daphnia magna exposed to neuro-active drugs. Science of the Total Environment, 662: 160-167, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, A. Sanfeliu and F. Moreno-Noguer. Online learning and detection of faces with low human supervision. The Visual Computer, 35(3): 349-370, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, C. Torras and G. Alenyà. Practical resolution methods for MDPs in robotics exemplified with disassembly planning. IEEE Robotics and Automation Letters, 4(3): 2282-2288, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. A scalable, efficient, and accurate solution to non-rigid structure from motion. Computer Vision and Image Understanding, 167: 121-133, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, G. Alenyà, J. Hernández-Farigola and C. Torras. Deciding the different robot roles for patient cognitive training. International Journal of Human-Computer Studies, 117: 20-29, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10313-10322, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà, G. Pons-Moll and F. Moreno-Noguer. SMPLicit: Topology-aware generative model for clothed people, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 11870-11880, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Garcia, A. Agudo and F. Moreno-Noguer. E-DNAS: Differentiable neural architecture search for embedded systems, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 4704-4711.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Lundell, E. Corona, T. Nguyen Le, F. Verdoja, P. Weinzaepfel, G. Rogez, F. Moreno-Noguer and V. Kyrki. Multi-FinGAN: Generative coarse-to-fine sampling of multi-finger grasps, 2021 IEEE International Conference on Robotics and Automation, 2021, Xian, China, pp. 4495-4501.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Total estimation from RGB video: On-line camera self-calibration, non-rigid shape and motion, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 8140-8147.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, J. Segovia, C. Torras and G. Alenyà. Online action recognition, 35th AAAI Conference on Artificial Intelligence, 2021, (Virtual), pp. 11981-11989.

    Open/Close abstract Abstract Info Info pdf PDF

  • W. Guo, E. Corona, F. Moreno-Noguer and X. Alameda. PI-Net: Pose interacting network for multi-person monocular 3D pose estimation, 21st IEEE Winter Conference on Applications of Computer Vision, 2021, (Virtual), pp. 2795-2805.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Piecewise Bézier space: Recovering 3D dynamic motion from video, 28th IEEE International Conference on Image Processing, 2021, Anchorage, AK, USA, pp. 3268-3272.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Vakhitov, L. Ferraz, A. Agudo and F. Moreno-Noguer. Uncertainty-Aware Camera Pose Estimation from Points and Lines, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 4657-4666, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, R. Huertas, S. Forgas, C. Torras and G. Alenyà. Discovering SOCIABLE: Using a conceptual model to evaluate the legibility and effectiveness of backchannel cues in an entertainment scenario, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Naples, Italy (Virtual), pp. 752-759.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Sanchez and F. Moreno-Noguer. Integrating human body mocaps into Blender using RGB images, 13th International Conference on Advances in Computer-Human Interactions, 2020, Valencia, pp. 285-290.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Maceira, A. Olivares-Alarcos and G. Alenyà. Recurrent neural networks for inferring intentions in shared tasks for industrial collaborative robots, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Naples, Italy (Virtual), pp. 665-670.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Geer, M. Maceira, J. Borràs, C. Torras and G. Alenyà. Interaction identification through tactile sensing during cloth manipulation using a 3-axis touch sensor, 2020 IROS Workshop on Robotic Manipulation of Deformable Objects, 2020, (Virtual), pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Textual visual semantic dataset for Text Spotting, 2020 CVPR Workshop on Text and Documents in the Deep Learning Era, 2020, Seattle, WA, USA, pp. 2306-2315, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Jangir, G. Alenyà and C. Torras. Dynamic cloth manipulation with deep reinforcement learning, 2020 IEEE International Conference on Robotics and Automation, 2020, Paris, France, pp. 4630-4636, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, T. Gaugry, J. Segovia, A. Bernardin, C. Torras, M. Marchal and G. Alenyà. Leveraging multiple environments for learning and decision making: a dismantling use case, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020, Las Vegas, NV, USA, pp. 6902-6908.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà and F. Moreno-Noguer. Context-aware human motion prediction, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA (Virtual), pp. 6990-6999.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, J. Segovia, C. Torras and G. Alenyà. STRIPS action discovery, 2020 AAAI 2020 workshop on Generalization in Planning, 2020, New York, pp. 1-9.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà, F. Moreno-Noguer and G. Rogez. GanHand: Predicting human grasp affordances in multi-object scenes, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA (Virtual), pp. 5030-5040.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Segmentation and 3D reconstruction of non-rigid shape from RGB video, 27th IEEE International Conference on Image Processing, 2020, Abu Dhabi, United Arab Emirates (Virtual), pp. 2845-2849.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Sidhu, E. Tretschk, V. Golyanik, A. Agudo and C. Theobalt. Neural dense non-rigid structure from motion with latent space constraints, 16th European Conference on Computer Vision, 2020, (Virtual), in Computer Vision - ECCV 2020, Vol of Lecture Notes in Computer Science, pp. 12361:204-222, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, M. Cashmore, S. Krivić, G. Alenyà, D. Magazzeni and C. Torras. Probabilistic planning for robotics with ROSPlan, 2019 Annual Conference Towards Autonomous Robotic Systems Conference, 2019, London, in Towards Autonomous Robotic Systems, Vol 11649 of Lecture Notes in Computer Science, pp. 236-250, 2019, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Learning robot policies using a high-level abstraction persona-behaviour simulator, 28th IEEE International Symposium on Robot and Human Interactive Communication, 2019, New Delhi, India, pp. 1-8.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, A. Suárez, J. Segovia, C. Torras and G. Alenyà. Natural teaching of robot-assisted rearranging exercises for cognitive training, 11th International Conference on Social Robotics, 2019, Madrid, in Social Robotics, Vol 11876 of Lecture Notes in Computer Science, pp. 611-621, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, J. Sanchez, G. P. T. Choi, A. Sanfeliu and F. Moreno-Noguer. 3DPeople: Modeling the geometry of dressed humans, 2019 International Conference on Computer Vision, 2019, Seoul, pp. 2242-2251.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Hernandez Ruiz, J. Gall and F. Moreno-Noguer. Human motion prediction via spatio-temporal inpainting, 2019 International Conference on Computer Vision, 2019, Seoul, pp. 7133-7142.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Semantic relatedness based re-ranker for text spotting, 14th Conference on Empirical Methods in Natural Language Processing, 2019, Hong Kong.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Detailed 3D face reconstruction from a single RGB image, 27th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, 2019, Plzen, Czech Republic, Vol 27(2) of Journal of WSCG, pp. 103-112.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Single view facial hair 3D reconstruction, 2019 Iberian Conference on Pattern Recognition and Image Analysis, 2019, Madrid, Spain, pp. 423-436.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Lumbreras, F. Moreno-Noguer and A. Agudo. 2D-to-3D facial expression transfer, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 2008-2013.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Deformable motion 3D reconstruction by union of regularized subspaces, 25th IEEE International Conference on Image Processing, 2018, Athens, Greece, pp. 2930-2934.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: Anatomically-aware facial animation from a single image, 15th European Conference on Computer Vision, 2018, Munich, Germany, in Computer Vision - ECCV 2018, Vol 11214 of Lecture Notes in Computer Science, pp. 835-851, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, G. Alenyà and C. Torras. Interleaving hierarchical task planning and motion constraint testing for dual-arm manipulation, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, Spain, pp. 4061-4066.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Hallucinating dense optical flow from sparse lidar for autonomous vehicles, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 1959-1964.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Vila, G. Canal and G. Alenyà. Towards safety in physically assistive robots: eating assistance, 2018 IROS Workshop on Robots for Assisted Living, 2018, Madrid, pp. 1-4.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Visual re-ranking with natural language understanding for text spotting, 14th Asian Conference on Computer Vision, 2018, Perth, in Computer Vision – ACCV 2018, Vol 11363 of Lecture Notes in Computer Science, pp. 68-82, 2019, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Deep lidar CNN to understand the dynamics of moving vehicles, 2018 IEEE International Conference on Robotics and Automation, 2018, Brisbane, Australia, pp. 4504-4509.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, M. Pijoan and F. Moreno-Noguer. Image collection pop-up: 3D reconstruction and clustering of rigid and non-rigid categories, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, UT, USA, pp. 2607-2615.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Visual semantic re-ranker for text spotting, 23rd Iberoamerican Congress on Pattern Recognition, 2018, Madrid, in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Vol 11401 of Lecture Notes in Computer Science, pp. 884-892, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit and F. Moreno-Noguer. Geometry-aware network for non-rigid shape prediction from a single view, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, UT, USA, pp. 4681-4690.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Unsupervised person image synthesis in arbitrary poses, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, UT, USA, pp. 8620-8628.

    Open/Close abstract Abstract Info Info pdf PDF