Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2023 - 2022) of the Perception and Manipulation

  • M. Arduengo, A. Colomé, J. Lobo, L. Sentis and C. Torras. Gaussian-process-based learning from demonstration. Journal of Ambient Intelligence and Humanized Computing, 2023, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. Social robot-delivered customer-facing services: an assessment of the experience. The Service Industries Journal, 43(3): 154-184, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Amadio, J.A. Delgado-Guerrero, A. Colomé and C. Torras. Controlled Gaussian process dynamical models with application to robotic cloth manipulation. International Journal of Dynamics and Control , 2023, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs, A. Boix, S. Foix and C. Torras. A virtual reality framework for fast dataset creation applied to cloth manipulation with automatic semantic labelling, 2023 IEEE International Conference on Robotics and Automation, 2023, London (UK), pp. 11605-11611.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Detail-aware uncalibrated photometric stereo, 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, Rhodes Island, Greece, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • D.F. Ordoñez, M. Martin, A. Agudo and F. Moreno-Noguer. On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis, 2023 Robotics: Science and Systems Conference, 2023, Daegu, Republic of Korea.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro, J. Fontana, J. Amorós, M. Alberich-Carramiñana, J. Borràs and C. Torras. A representation of cloth states based on a derivative of the Gauss Linking Integral. Applied Mathematics and Computation, 457: 128165, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, A. Andriella, S. Foix and G. Alenyà. Robot explanatory narratives of collaborative and adaptive experiences, 2023 IEEE International Conference on Robotics and Automation, 2023, London (UK), pp. 11964-11971.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro, J. Amorós, M. Alberich-Carramiñana and C. Torras. Reconstruction of sampled surfaces with boundary via Morse theory, 2023 Congreso Español de Informática Gráfica, 2023, Mallorca, pp. 21-24.

    Open/Close abstract Abstract Info Info pdf PDF

  • E.B. Bueno Benito, B. Tura and M. Dimiccoli. Leveraging triplet loss for unsupervised action segmentation, 2023 CVPR Workshop on Learning with Limited Labelled Data, 2023, Vancouver, Canadá, pp. 4922-4930, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Robótica asistencial. Retos éticos en la ayuda y acompañamiento a cuidadores y pacientes. In Soledad no deseada en la era digital, 38-46. Fundació Víctor Grífols i Lucas, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. Gendered human–robot interactions in services. International Journal of Social Robotics, 2023, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras and L.G. Ludescher. Writing science fiction as an inspiration for AI research and ethics dissemination. In Human-Centered Artificial Intelligence: Advanced Lectures, 322-344. Cham: Springer International Publishing, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras, C. Abdelnour and G. Alenyà. Introducing CARESSER: A framework for in situ learning robot social assistance from expert knowledge and demonstrations. User Modeling and User-Adapted Interaction, 33: 441-496, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, S. Foix and G. Alenyà. Time to contact for robot safety stop in close collaborative tasks. In Human-robot collaboration: unlocking the potential for industrial applications, 87-104. Institution of Engineering Technology, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, C. Torras and G. Alenyà. Generating predicate suggestions based on the space of plans: an example of planning with preferences. User Modeling and User-Adapted Interaction, 33: 333-357, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Pérez, A. Espersen and A. Agudo. Robust wind turbine blade segmentation from RGB images in the wild, 2023 IEEE International Conference on Image Processing, 2023, Kuala Lumpur, Malaysia, pp. 1025-1029.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. Does the personality of consumers influence the assessment of the experience of interaction with social robots?. International Journal of Social Robotics, 2023, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, V. Puig and G. Alenyà. Condition-based design of variable impedance controllers from user demonstrations. Control Engineering Practice, 140: 105658, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, S. Foix, S. Borgo and G. Alenyà. OCRA – An ontology for collaborative robotics and adaptation. Computers in Industry, 132: 103627, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Spline human motion recovery, 2022 IEEE International Conference on Image Processing, 2022, Bordeaux, France, pp. 4138-4142, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Cervells artificials: ciència o ficció?. In Cervell(s), 48-55. Centre de Cultura Contemporània de Barcelona (CCCB), 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs, B. Calli, A. Norton and G. Alenyà. Household cloth object set: Fostering benchmarking in deformable object manipulation. IEEE Robotics and Automation Letters, 7(3): 5866-5873, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras and L. Delicado. Blanco y frío como el Polo Norte y el Polo Sur: Seres humanos y robots. Sociology and Technoscience, 12(1): 257-275, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • G.D. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer and G. Rogez. PoseScript: 3D human poses from natural language, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 346-362, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Izquierdo, G. Canal, C. Rizzo and G. Alenyà. Improved task planning through failure anticipation in human-robot collaboration, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 7875-7880.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Caldarelli, A. Colomé and C. Torras. Perturbation-Based stiffness inference in variable impedance control. IEEE Robotics and Automation Letters, 7(4): 8823-8830, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. La ciencia ficción como estímulo del debate ético en robótica. In Robótica, ética y política. El impacto de la superinteligencia en el mundo de las personas, 139-167. Icaria Editorial, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, G. Alenyà and V. Puig. Automated off-line generation of stable variable impedance controllers according to performance specifications. IEEE Robotics and Automation Letters, 7(3): 5874-5881, 2022, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dimiccoli, S. Patni, M. Hoffmann and F. Moreno-Noguer. Recognizing object surface material from impact sounds for robot manipulation, 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022, Kyoto, pp. 9280-9287.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Alvarez, L. Arroyo, R. Avila, E. Crusellas, M. de la Fuente, L. Martínez, M. Mata, T. Ruiz, M. Ruiz and C. Torras. Les dades obertes i la intel·ligència artificial, eines per a la igualtat de gènere. Volume 7 of Govern Obert. Generalitat de Catalunya, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Orti, F. Moreno-Noguer and V. Puig. Guided-crop image augmentation for small defect classification, 26th International Conference on Pattern Recognition, 2022, Montreal, Canada, pp. 104-110.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Rossi, A. Andriella, S. Rossi, C. Torras and G. Alenyà. Evaluating the effect of Theory of Mind on people's trust in a faulty robot, 31st IEEE International Symposium on Robot and Human Interactive Communication, 2022, Napoli, Italy, pp. 477-482.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Belchí and A. Stefanou. A∞ Persistent Homology Estimates Detailed Topology from Pointcloud Datasets. Discrete & Computational Geometry, 68: 274-297, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs. Effective grasping enables successful robot-assisted dressing. Science robotics, 7(65): eabo7229, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto. Social assistive robotics: An ethical approach through the concept of freedom, 2022 Philosophy of Human-Technology Relations, 2022, Copenhagen, pp. 53.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, A. Andriella, G. Alenyà and C. Torras. Proactive learning of cognitive exercises with a social robot, 2022 ICRA Workshop on Prediction and Anticipation Reasoning in Human Robot Interaction (PAR-HRI), 2022, Philadelphia, USA, pp. 1-4.

    Open/Close abstract Abstract Info Info pdf PDF

  • D.F. Ordoñez, A. Agudo, F. Moreno-Noguer and M. Martin. An adaptable approach to learn realistic legged locomotion without examples, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 4671-4678.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto. Robótica social asistencial. Implicaciones y desafíos éticos. Brains. Business, Research, Ageing, Innovation, Neurosciences , 2(2): 39-43, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, G. Pons-Moll, G. Alenyà and F. Moreno-Noguer. Learned Vertex Descent: a new direction for 3D human model fitting, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 146--165, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 519-532, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Xu, A. Colomé, L. Sentis and C. Torras. Mixtures of controlled Gaussian processes for dynamical modeling of deformable objects, 2022 Learning for Dynamics & Control Conference, 2022, Stanford, CA, USA, pp. 415-426.

    Open/Close abstract Abstract Info Info pdf PDF

  • B.M. Lopes, R. Marques , T. Santos , J. Sousa and G. Alenyà. Relevant parameters identification in traditional & stretch and blow thermoplastics injection molding, 10th IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, 2022, Chemnitz, Germany , pp. 1-6, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Izquierdo, C. Rizzo and G. Alenyà. Planning interactions as an event handling solution for successful and balanced human-robot collaboration, 2022 IROS Workshop on Artificial Intelligence for Social Robots Interacting with Humans in the Real World, 2022, Kyoto, Japan, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro, J. Amorós, M. Alberich-Carramiñana and C. Torras. An inextensible model for the robotic manipulation of textiles. Applied Mathematical Modelling, 101: 832-858, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. The effects of gender and personality of robot assistants on customers’ acceptance of their service. Service Bussiness, 16: 359–389, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Canas, A.M. Garcia, C. Angulo and A. Garrell Zulueta. Initial test of “BabyRobot” behaviour on a teleoperated toy substitution: Improving the motor skills of toddlers, 2022 ACM/IEEE International Conference on Human-Robot Interaction, 2022, Sapporo, Japan, pp. 708-712.

    Open/Close abstract Abstract Info Info pdf PDF

  • L. González, C. Angulo, J.A. Ortega and J.L. López. Statistical validation of synthetic data for lung cancer patients generated by using generative adversarial networks. Electronics, 11(20): 3277, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, V. Lepetit and F. Moreno-Noguer. Simultaneous completion and spatiotemporal grouping of corrupted motion tracks. The Visual Computer, 38(11): 3937–3952, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Laplaza, A. Garrell Zulueta, F. Moreno-Noguer and A. Sanfeliu. Context and intention for 3D human motion prediction: Experimentation and user study in handover tasks, 31st IEEE International Symposium on Robot and Human Interactive Communication, 2022, Napoli, Italy, pp. 630-635.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Peral, A. Sanfeliu and A. Garrell Zulueta. Efficient hand gesture recognition for human-robot interaction. IEEE Robotics and Automation Letters, 7(4): 10272-10279, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Strazzeri and Rubén J. Sánchez-García. Possibility results for graph clustering: A novel consistency axiom. Pattern Recognition, 128: 108687, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Forgas, R. Huertas, A. Andriella and G. Alenyà. How do consumers' gender and rational thinking affect the acceptance of entertainment social robots?. International Journal of Social Robotics, 14: 973–994, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Estevez and A. Agudo. Uncalibrated, unified and unsupervised specular-aware photometric stereo, 2022 ICPR Workshop on Towards a Complete Analysis of People: From Face and Body to Clothes, 2022, Montreal (Canada), pp. 7-20.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Boix, S. Foix and C. Torras. Garment manipulation dataset for robot learning by demonstration through a virtual reality framework, 24th Catalan Conference on Artificial Intelligence, 2022, Sitges, in Artificial Intelligence Research and Development, Vol 356 of Frontiers in Artificial Intelligence and Applications, pp. 199-208, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Shen, A. Agudo, F. Moreno-Noguer and A. Ruiz. Conditional-Flow NeRF: Accurate 3D modelling with reliable uncertainty quantification, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 540-557, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • X. Xu, H. Chen, F. Moreno-Noguer, L. Jeni and F. De la Torre. 3D human pose, shape and texture from low-resolution images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs and G. Alenyà. Knowledge representation to enable high-level planning in cloth manipulation tasks, 2022 ICAPS Workshop on Knowledge Engineering for Planning and Scheduling, 2022, Singapore (Virtual), pp. 9.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, R. Huertas, S. Forgas, C. Torras and G. Alenyà. “I know how you feel” The importance of interaction style on users’ acceptance in an entertainment scenario. Interaction Studies: Social Behaviour and Communication in Biological and Artificial Systems, 23(1): 21-57, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • F.J. Moreno, V.J. Traver, F. Barranco, M. Dimiccoli and F. Pla. Visual event-based egocentric human action recognition, 2022 Iberian Conference on Pattern Recognition and Image Analysis, 2022, Aveiro, Portugal, Vol 13256 of Lecture Notes in Computer Science, pp. 402-404, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pérez and A. Agudo. Matching and recovering 3D people from multiple views, 2022 IEEE Winter Conference on Applications of Computer Vision, 2022, Waikoloa, Hawaii, USA, pp. 1184-1193, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs, B. Calli, A. Norton and G. Alenyà. Cloth manipulation and perception competition, 2022 ICRA Workshop on Representing and Manipulating Deformable Objects, 2022, Philadelphia, pp. 4.

    Open/Close abstract Abstract Info Info pdf PDF

  • N. Ugrinovic, A. Pumarola, A. Sanfeliu and F. Moreno-Noguer. Single-view 3d body and cloth reconstruction under complex poses, 17th International Conference on Computer Vision Theory and Applications, 2022, Online.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. History of the Institut de Robòtica i Informàtica Industrial, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Safari from visual signals: Recovering volumetric 3D shapes, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, Singapore, pp. 2495-2499.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Amorós, F. Coltraro, C. Torras and M. Verdaguer. Morse cell decomposition and parametrization of surfaces from point clouds, 2022 Encuentros de Álgebra Computacional y Aplicaciones, 2022, Castellón, Spain, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Ordinal inverse reinforcement learning applied to robot learning with small data, 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022, Kyoto, pp. 2490-2496.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Learning in autonomous and intelligent systems: Overview and biases from data sources. Arbor - Ciencia, Pensamiento y Cultura, 197(802): a627, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro, J. Amorós, M. Alberich-Carramiñana and C. Torras. An inextensible model for the robotic manipulation of textiles, 2022 Congreso de ecuaciones diferenciales y aplicaciones XXVII, 2022, Zaragoza, Spain, pp. 87-87.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Mutual information weighing for probabilistic movement primitives, 24th Catalan Conference on Artificial Intelligence, 2022, Sitges, in Artificial Intelligence Research and Development, Vol 356 of Frontiers in Artificial Intelligence and Applications, pp. 365-368, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF

  • W. Zhang, M. Dimiccoli and B.Y. Lim. Debiased-CAM to mitigate image perturbations with faithful visual explanations of machine learning , 2022 ACM Conference on Human Factors in Computing Systems, 2022, New Orleans, pp. 1-32, ACM.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto, B. Román and C. Torras. Ethics for social robotics: A critical analysis, 2022 The Road to a Successful HRI: AI, Trust and ethicS Workshop, in the ACM/IEEE International Conference on Human-Robot Interaction (HRI 2022), 2022, Sapporo, Japan, pp. 1-3.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Implications of robot backchannelling in cognitive therapy, 14th International Conference on Social Robotics, 2022, Florence, Italy, Vol 13817 of Lecture Notes in Computer Science, pp. 546–557.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is the main facility devoted to experimentation on the topics of the research line. It occupies 142 m² in the second floor of the Faculty, and a large part hosts a life-scale mock-up of a (quite) small apartment (35 m²). Two PAL one-armed mobile TIAGo robots dwell within the apartment, but are occasionally taken out for experimentation elsewhere. There is also a manipulation area equipped with two WAM robot arms and a KINOVA manipulator, and workplaces are distributed along the perimeter. Next to the laboratory, with a direct view through a window, there is the Scientific and Technical Lab support office.

The Laboratory is further equipped with commercial and self-developed grippers, sensing devices, and augmented reality appliances. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosted the Humanoids Lab Initiative in the past, and we still keep 15 small humanoid robots for educational or promotional purposes.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

TFG Students

Support Staff