Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2022 - 2021) of the Perception and Manipulation

  • F. Coltraro, J. Amorós, M. Alberich-Carramiñana and C. Torras. An inextensible model for the robotic manipulation of textiles. Applied Mathematical Modelling, 101: 832-858, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 519-532, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Strazzeri and C. Torras. Configuration space of a textile rectangle. In Geometry, Topology, Algebra, and Applications; Women in Geometry and Topology, 133-137. Springer-Birkhäuser, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • B. Nicolás, X. Wu, M. Dimiccoli, J. Sierpowska, C. Saiz-Masvidal, C. Soriano-Mas, P. Radeva and L. Fuentemilla. Behavioural and neurophysiological signatures in the retrieval of individual memories of recent and remote real-life routine episodic events. Cortex, 141: 128-143, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Carfi, T. Patten, Y. Kuang, A. Hammoud, M. Alameh, E. Maiettini, A.I. Weinberg, D. Faria, F. Mastrogiovanni, G. Alenyà, L. Natale, V. Perdereau, M. Vincze and A. Billard. Hand-Object Interaction: From Human Demonstrations to Robot Manipulation. Frontiers in Robotics and AI, 8: 714023, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Garcia, A. Agudo and F. Moreno-Noguer. E-DNAS: Differentiable neural architecture search for embedded systems, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 4704-4711.

    Open/Close abstract Abstract Info Info pdf PDF

  • M.A. Roa, M.R. Dogar, J. Pagès, C. Vivas, A. Morales, N. Correll, M. Görner, J. Rosell, S. Foix, R. Memmesheier and F. Ferro. Mobile manipulation hackathon: moving into real world applications. IEEE Robotics and Automation Magazine, 28(2): 112-124, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, A. Andriella, A. Taranović, J. Segovia, C. Torras and G. Alenyà. Automatic learning of cognitive exercises for socially assistive robotics, 30th IEEE International Symposium on Robot and Human Interactive Communication, 2021, Vancouver, Canada, pp. 139-146.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Total estimation from RGB video: On-line camera self-calibration, non-rigid shape and motion, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 8140-8147.

    Open/Close abstract Abstract Info Info pdf PDF

  • N.H. , R. Jangir, Y.S. , G. Alenyà, P. Abbeel, A.A. , L.P. and X.W.. Self-supervised policy adaptation during deployment, 9th International Conference on Learning Representations, 2021, (Virtual).

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Vakhitov, L. Ferraz, A. Agudo and F. Moreno-Noguer. Uncertainty-Aware Camera Pose Estimation from Points and Lines, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 4657-4666, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, P. Radeva and M. Dimiccoli. Modeling long-term interactions to enhance action recognition, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 10351-10358.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Piecewise Bézier space: Recovering 3D dynamic motion from video, 28th IEEE International Conference on Image Processing, 2021, Anchorage, AK, USA, pp. 3268-3272.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, C. Torras and L. Sentis. Robust and adaptive door operation with a mobile robot. Intelligent Service Robotics, 14(3): 409-425, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, H. Siqueira, D. Fu, S. Magg, P. Barros, S. Wermter, C. Torras and G. Alenyà. Do I have a personality? Endowing care robots with context-dependent personality traits. International Journal of Social Robotics, 13(8): 2081-2102, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Felicioni and M. Dimiccoli. Interaction-GCN: a Graph Convolutional Network based framework for social interaction recognition in egocentric videos, 28th IEEE International Conference on Image Processing, 2021, Anchorage, AK, USA, pp. 2348-2352.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. How do consumers' gender and rational thinking affect the acceptance of entertainment social robots?. International Journal of Social Robotics, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, E. Talavera, P. Radeva and M. Dimiccoli. Understanding event boundaries for egocentric activity recognition from photo-streams, 2021 ICPR International Workshops and Challenges, 2021, Milan, Italy (Virtual), in Pattern Recognition. ICPR International Workshops and Challenges. Proceedings, Part III, Vol 12663 of Lecture Notes in Computer Science, pp. 334-347, 2021, Springer International Publishing.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, V. Lepetit and F. Moreno-Noguer. Simultaneous completion and spatiotemporal grouping of corrupted motion tracks. The Visual Computer, 2021, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Ciència i ficció: quina inspira quina?. In ENTRE LA MANO Y LA CABEZA: Trece puntos de encuentro entre la ciencia y la literatura, 117-125. Inspiraciencia, CSIC Catalunya, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ruiz and J. Verbeek. Anytime inference with distilled hierarchical neural ensembles, 35th AAAI Conference on Artificial Intelligence, 2021, (Virtual), pp. 9463-9471.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, A. Arduengo, A. Colomé, J. Lobo and C. Torras. Human to robot whole-body motion transfer, 2020 IEEE-RAS International Conference on Humanoid Robots, 2021, Munich, Germany (Virtual), pp. 299-305.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Sanchez, A. Pumarola and F. Moreno-Noguer. PhysXNet: A customizable approach for learning cloth dynamics on dressed people, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 879-888.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ruiz, A. Agudo and F. Moreno-Noguer. Generating attribution maps with disentangled masked backpropagation, 2021 International Conference on Computer Vision, 2021, Virtual, pp. 905-914, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dimiccoli, L. Garrido, G. Rodríguez and H. Wendt. Graph constrained data representation learning for human motion segmentation, 2021 International Conference on Computer Vision, 2021, Virtual, pp. 1460-1469, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • X. Xu, H. Chen, F. Moreno-Noguer, L. Jeni and F. De la Torre. 3D human pose, shape and texture from low-resolution images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, J. Segovia, C. Torras and G. Alenyà. Online action recognition, 35th AAAI Conference on Artificial Intelligence, 2021, (Virtual), pp. 11981-11989.

    Open/Close abstract Abstract Info Info pdf PDF

  • B.M. Lopes, J. Sousa and G. Alenyà. Data acquisition and monitoring system for legacy injection machines, 9th IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, 2021, Hong Kong, China (Virtual).

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Ramon, G. Triginer, J. Escur, A. Pumarola, J. García, X. Giro-i-Nieto and F. Moreno-Noguer. H3D-Net: Few-shot high-fidelity 3D head reconstruction, 2021 International Conference on Computer Vision, 2021, Virtual, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto. Prolegómenos a una ética para la robótica social. DILEMATA, Revista Internacional de Éticas Aplicadas(34): 71-87, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dimiccoli, H. Wendt and P. Battle. Learning grounded word meaning representations on similarity graphs, 16th Conference on Empirical Methods in Natural Language Processing, 2021, Punta Cana, Dominican Republic, pp. 4760--4769.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Hernandez Ruiz, A. Vilalta and F. Moreno-Noguer. Neural Cellular Automata manifold, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10015-10023, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • W. Guo, E. Corona, F. Moreno-Noguer and X. Alameda. PI-Net: Pose interacting network for multi-person monocular 3D pose estimation, 21st IEEE Winter Conference on Applications of Computer Vision, 2021, (Virtual), pp. 2795-2805.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dalmasso, A. Garrell Zulueta, J.E. Domínguez, P. Jiménez and A. Sanfeliu. Human-robot collaborative multi-agent path planning using Monte Carlo tree search and Social Reward Sources, 2021 IEEE International Conference on Robotics and Automation, 2021, Xian, China, pp. 10133-10138.

    Open/Close abstract Abstract Info Info pdf PDF

  • N. Ugrinovic, A. Ruiz, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Body size and depth disambiguation in multi-person reconstruction from single images, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 53-63.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Shen, A. Ruiz, A. Agudo and F. Moreno-Noguer. Stochastic Neural Radiance Fields: Quantifying uncertainty in implicit 3D representations, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 972-981.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Laplaza, A. Pumarola, F. Moreno-Noguer and A. Sanfeliu. Attention deep learning based model for predicting the 3D human body pose using the robot human handover phases, 30th IEEE International Symposium on Robot and Human Interactive Communication, 2021, Vancouver, Canada, pp. 161-166.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10313-10322, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Simarro, D. Calvete, T. Plomaritis, F. Moreno-Noguer, I. Giannoukakou, J. Montes and R. Duran. The influence of camera calibration on nearshore bathymetry estimation from UAV videos. Remote Sensing, 13(1): 150, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto, B. Román and C. Torras. The ethical issues of social assistive robotics: A critical literature review. Technology in Society, 67: 101726, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà, G. Pons-Moll and F. Moreno-Noguer. SMPLicit: Topology-aware generative model for clothed people, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 11870-11880, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, A. Colomé, J. Borràs, L. Sentis and C. Torras. Task-adaptive robot learning from demonstration with Gaussian process models under replication. IEEE Robotics and Automation Letters, 6(2): 966-973, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs and G. Alenyà. Benchmarking cloth manipulation using action graphs: an example in placing flat, 1st IROS Workshop on Benchmarking of robotic grasping and manipulation: protocols, metrics and data analysis, 2021, Prague, Czech Republic (Visrtual), pp. 1-3, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Chatziagapi, S. Athar, F. Moreno-Noguer and D. Samaras. SIDER: Single-image neural optimization for facial geometric detail recovery, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 815-824.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. A topological extension of movement primitives for curvature modulation and sampling of robot motion. Autonomous Robots, 45: 725-735, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, C. Torras and G. Alenyà. Are preferences useful for better assistance?: A physically assistive robotics user study. ACM Transactions on Human-Robot Interaction, 10(4): 1-19, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Lundell, E. Corona, T. Nguyen Le, F. Verdoja, P. Weinzaepfel, G. Rogez, F. Moreno-Noguer and V. Kyrki. Multi-FinGAN: Generative coarse-to-fine sampling of multi-finger grasps, 2021 IEEE International Conference on Robotics and Automation, 2021, Xian, China, pp. 4495-4501.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Dual-branch CNNs for vehicle detection and tracking on LiDAR data. IEEE Transactions on Intelligent Transportation Systems, 22(11): 6942-6953, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Strazzeri and C. Torras. Topological representation of cloth state for robot manipulation. Autonomous Robots, 45: 737–754, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, V. Puig and G. Alenyà. Disturbance observer-based LPV feedback control of a N-DoF robotic manipulator including compliance through gain shifting. Control Engineering Practice, 115(104887), 2021.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is the main facility devoted to experimentation on the topics of the research line. It occupies 142 m² in the second floor of the Faculty, and a large part hosts a life-scale mock-up of a (quite) small apartment (35 m²). Two PAL one-armed mobile TIAGo robots dwell within the apartment, but are occasionally taken out for experimentation elsewhere. There is also a manipulation area equipped with two WAM robot arms and a KINOVA manipulator, and workplaces are distributed along the perimeter. Next to the laboratory, with a direct view through a window, there is the Scientific and Technical Lab support office.

The Laboratory is further equipped with commercial and self-developed grippers, sensing devices, and augmented reality appliances. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosted the Humanoids Lab Initiative in the past, and we still keep 15 small humanoid robots for educational or promotional purposes.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

TFG Students

Support Staff