Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2021 - 2020) of the Perception and Manipulation

  • J. Garcia, A. Agudo and F. Moreno-Noguer. E-DNAS: Differentiable neural architecture search for embedded systems, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 4704-4711.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Piecewise Bézier space: Recovering 3D dynamic motion from video, 2021 IEEE International Conference on Image Processing, 2021, Anchorage (Alaska, USA), pp. 3268-3272.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Total estimation from RGB video: On-line camera self-calibration, non-rigid shape and motion, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 8140-8147.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Felicioni and M. Dimiccoli. Interaction-GCN: a Graph Convolutional Network based framework for social interaction recognition in egocentric videos, 2021 IEEE International Conference on Image Processing, 2021, Anchorage (Alaska, USA), pp. 2348-2352.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, P. Radeva and M. Dimiccoli. Modeling long-term interactions to enhance action recognition, 25th International Conference on Pattern Recognition, 2021, Milan, Italy (Virtual), pp. 10351-10358.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, V. Lepetit and F. Moreno-Noguer. Simultaneous completion and spatiotemporal grouping of corrupted motion tracks. The Visual Computer, 2021, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, H. Siqueira, D. Fu, S. Magg, P. Barros, S. Wermter, C. Torras and G. Alenyà. Do I have a personality? Endowing care robots with context-dependent personality traits. International Journal of Social Robotics, 2021, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, A. Arduengo, A. Colomé, J. Lobo and C. Torras. Human to robot whole-body motion transfer, 2020 IEEE-RAS International Conference on Humanoid Robots, 2021, Munic (Online), to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, E. Talavera, P. Radeva and M. Dimiccoli. Understanding event boundaries for egocentric activity recognition from photo-streams, 2021 ICPR International Workshops and Challenges, 2021, Milan, Italy (Virtual), in Pattern Recognition. ICPR International Workshops and Challenges. Proceedings, Part III, Vol 12663 of Lecture Notes in Computer Science, pp. 334-347, 2021, Springer International Publishing.

    Open/Close abstract Abstract Info Info pdf PDF

  • B.M. Lopes, J. Sousa and G. Alenyà. Data acquisition and monitoring system for legacy injection machines, 2021 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications 2021, 2021, Hong Kong.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ruiz and J. Verbeek. Anytime inference with distilled hierarchical neural ensembles, 2021 AAAI Conference on Artificial Intelligence, 2021, Virtual, pp. 9463-9471.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Laplaza, A. Pumarola, F. Moreno-Noguer and A. Sanfeliu. Attention deep learning based model for predicting the 3D human body pose using the robot human handover phases, 2021 IEEE International Symposium on Robot and Human Interactive Communication, 2021, Vancouver (Canada), pp. 161-166.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, J. Segovia, C. Torras and G. Alenyà. Online action recognition, 2021 AAAI Conference on Artificial Intelligence, 2021, Virtual, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto, B. Román and C. Torras. The ethical issues of social assistive robotics: A critical literature review. Technology in Society, 67: 101726, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Pareto. Prolegómenos a una ética para la robótica social. DILEMATA, Revista Internacional de Éticas Aplicadas(34): 71-87, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs and G. Alenyà. Benchmarking cloth manipulation using action graphs: an example in placing flat, 2021 IROS Workshop on Benchmarking of robotic grasping and manipulation: protocols, metrics and data analysis, 2021, Online, pp. 1-3, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • W. Guo, E. Corona, F. Moreno-Noguer and X. Alameda. PI-Net: Pose interacting network for multi-person monocular 3D pose estimation, 2021 IEEE Winter Conference on Applications of Computer Vision, 2021, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, C. Torras and G. Alenyà. Are preferences useful for better assistance?: A physically assistive robotics user study. ACM Transactions on Human-Robot Interaction, 10(4): 1-19, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Simarro, D. Calvete, T. Plomaritis, F. Moreno-Noguer, I. Giannoukakou, J. Montes and R. Duran. The influence of camera calibration on nearshore bathymetry estimation from UAV videos. Remote Sensing, 13(1): 150, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, V. Puig and G. Alenyà. Disturbance observer-based LPV feedback control of a N-DoF robotic manipulator including compliance through gain shifting. Control Engineering Practice, 115(104887), 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, A. Colomé, J. Borràs, L. Sentis and C. Torras. Task-adaptive robot learning from demonstration with Gaussian process models under replication. IEEE Robotics and Automation Letters, 6(2): 966-973, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Carfi, T. Patten, Y. Kuang, A. Hammoud, M. Alameh, E. Maiettini, A.I. Weinberg, D. Faria, F. Mastrogiovanni, G. Alenyà, L. Natale, V. Perdereau, M. Vincze and A. Billard. Hand-Object Interaction: From Human Demonstrations to Robot Manipulation. Frontiers in Robotics and AI, 8: 714023, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. A topological extension of movement primitives for curvature modulation and sampling of robot motion. Autonomous Robots, 45: 725-735, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Strazzeri and C. Torras. Topological representation of cloth state for robot manipulation. Autonomous Robots, 45: 737–754, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • B. Nicolás, X. Wu, M. Dimiccoli, J. Sierpowska, C. Saiz-Masvidal, C. Soriano-Mas, P. Radeva and L. Fuentemilla. Behavioural and neurophysiological signatures in the retrieval of individual memories of recent and remote real-life routine episodic events. Cortex, 141: 128-143, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M.A. Roa, M.R. Dogar, J. Pagès, C. Vivas, A. Morales, N. Correll, M. Görner, J. Rosell, S. Foix, R. Memmesheier and F. Ferro. Mobile manipulation hackathon: moving into real world applications. IEEE Robotics and Automation Magazine, 28(2): 112-124, 2021.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Dual-branch CNNs for vehicle detection and tracking on LiDAR data. IEEE Transactions on Intelligent Transportation Systems, 2021, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • N.H. , R. Jangir, Y.S. , G. Alenyà, P. Abbeel, A.A. , L.P. and X.W.. Self-supervised policy adaptation during deployment, Ninth International Conference on Learning Representations, 2021, Online.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs, G. Alenyà and C. Torras. A grasping-centered analysis for cloth manipulation. IEEE Transactions on Robotics, 36(3): 924-936, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Maceira, A. Olivares-Alarcos and G. Alenyà. Recurrent neural networks for inferring intentions in shared tasks for industrial collaborative robots, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Naples, Italy (Virtual), pp. 665-670.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Jangir, G. Alenyà and C. Torras. Dynamic cloth manipulation with deep reinforcement learning, 2020 IEEE International Conference on Robotics and Automation, 2020, Paris, France, pp. 4630-4636, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro. Experimental validation of an inextensible cloth model. Technical Report IRI-TR-20-04, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà and F. Moreno-Noguer. Context-aware human motion prediction, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA (Virtual), pp. 6990-6999.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Strazzeri and C. Torras. Configuration space of a textile rectangle. In Geometry of Varieties and Applications, . Springer-Birkhäuser, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà, F. Moreno-Noguer and G. Rogez. GanHand: Predicting human grasp affordances in multi-object scenes, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA (Virtual), pp. 5030-5040.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Geer, M. Maceira, J. Borràs, C. Torras and G. Alenyà. Interaction identification through tactile sensing during cloth manipulation using a 3-axis touch sensor, 2020 IROS Workshop on Robotic Manipulation of Deformable Objects, 2020, Online, pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, S. Popov, F. Moreno-Noguer and V. Ferrari. C-Flow: Conditional generative flow models for images and 3D point clouds, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA (Virtual), pp. 7946-7955.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dimiccoli and H. Wendt. Learning event representations for temporal segmentation of image sequences by dynamic graph embedding. IEEE Transactions on Image Processing, 30: 1476-1486, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot. In Fault Diagnosis and Fault-tolerant Control of Robotic Systems, . The Institution of Engineering and Technology, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Segmentation and 3D reconstruction of non-rigid shape from RGB video, 27th IEEE International Conference on Image Processing, 2020, Abu Dhabi, United Arab Emirates (Virtual), pp. 2845-2849.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Lobo, A. Enkaoua, A. Rodríguez-Fernández, N. Sharifrazi, J. Medina-Cantillo, J.M. Font, C. Torras and D.J. Reinkensmeyer. Evaluation of an exercise-enabling control interface for powered wheelchair users: a feasibility study with Duchenne muscular dystrophy. Journal of NeuroEngineering and Rehabilitation, 17(142), 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Segovia, S. Jiménez and A. Jonsson. Generalized planning with positive and negative examples, 34th AAAI Conference on Artificial Intelligence, 2020, New York, USA, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez and C. Torras. Perception of cloth in assistive robotic manipulation tasks. Natural Computing, 19(2): 409-431, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • V.A. Carmona, J. Lobo, J. van Ruysevelt, C. Torras and J.M. Font. Development and pilot evaluation of the ArmTracker: A wearable system to monitor arm kinematics during daily life, 8th IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics , 2020, New York City, NY, USA, pp. 759-764, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Short-term human-robot interaction adaptability in real-world environments. International Journal of Social Robotics, 12: 639-657, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Vendrell, C. Sun, V. Puig and G. Cembrano. Fuzzy clustering application on failure rate prediction in water distribution networks. Technical Report IRI-TR-20-02, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez, F. Dachs and V. González. Multiplicities of jumping points for mixed multiplier ideals. Revista Matematica Complutense, 33(1): 325-348, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128: 698-713, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Pignaton, J. Olszewska, J.L. Carbonera, S. Fiorini, A. Khamis, V.R. Kumar, M. Barreto, E. Prestes, M.K. Habib, S. Redfield, A. Chibani, P. Goncalves, J. Bermejo-Alonso, R. Sanz, E. Tosello, A. Olivares-Alarcos, A.A. Konzen, J. Quintas and H. Li. Ontological concepts for information sharing in cloud robotics. Journal of Ambient Intelligence and Humanized Computing: 1-12, 2020, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Liu, X. Li, J. Zhou and J. Shen. Prediction stability as a criterion in active learning, 2020 International Conference on Artificial Neural Networks, 2020, Bratislava, Slovakia, in Artificial Neural Networks and Machine Learning – ICANN 2020, Vol 12397 of Lecture Notes in Computer Science, pp. 157-167, 2020, Springer, Cham, Switzerland.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, M. Lippi, M.C. Welle, H. Yin, R. Antanova, A. Varava, J. Borràs, C. Torras, A. Marino, G. Alenyà and D. Kragic. Benchmarking bimanual cloth manipulation. IEEE Robotics and Automation Letters, 5(2): 1111-1118, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Sidhu, E. Tretschk, V. Golyanik, A. Agudo and C. Theobalt. Neural dense non-rigid structure from motion with latent space constraints, 16th European Conference on Computer Vision, 2020, Online, in Computer Vision - ECCV 2020, Vol of Lecture Notes in Computer Science, pp. 12361:204-222, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Textual visual semantic dataset for Text Spotting, 2020 CVPR Workshop on Text and Documents in the Deep Learning Era, 2020, Seattle, WA, USA, pp. 2306-2315, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Reinforcement Learning of Bimanual Robot Skills. Volume 134 of Springer Tracts in Advanced Robotics. Springer, 2020.

    Open/Close abstract Abstract Info Info

  • J.C. Gassó Loncan Vallecillo, A. Olivares-Alarcos and G. Alenyà. Visual feedback for humans about robots' perception in collaborative environments. Technical Report IRI-TR-20-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • X. Xu, H. Chen, F. Moreno-Noguer, L. Jeni and F. de la Torre. 3D human shape and pose from a single low-resolution image with self-supervised learning, 16th European Conference on Computer Vision, 2020, Online, in Computer Vision - ECCV 2020, Vol of Lecture Notes in Computer Science, pp. 12354:284-300, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Mitjans, M. Maceira and G. Alenyà. Corner detection of deformable fabric using deep learning. Technical Report IRI-TR-20-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, R. Huertas, S. Forgas, C. Torras and G. Alenyà. Discovering SOCIABLE: Using a conceptual model to evaluate the legibility and effectiveness of backchannel cues in an entertainment scenario, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Naples, Italy (Virtual), pp. 752-759.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Shi, E. Riba, D. Mishkin, F. Moreno-Noguer and A. Nicolaou. Differentiable data augmentation with Kornia, 1st Workshop on Differentiable Vision, Graphics, and Physics Applied to Machine Learning, 2020, Online.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Cognitive system framework for brain-training exercise based on human-robot interaction. Cognitive Computation, 12: 793-810, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Science-fiction: A mirror for the future of humankind. IDEES(48): 1-11, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, T. Gaugry, J. Segovia, A. Bernardin, C. Torras, M. Marchal and G. Alenyà. Leveraging multiple environments for learning and decision making: a dismantling use case, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020, Las Vegas, NV, USA, pp. 6902-6908.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Parent, A. Colomé and C. Torras. Variable impedance control in Cartesian latent space while avoiding obstacles in null space, 2020 IEEE International Conference on Robotics and Automation, 2020, Paris, France, pp. 9888-9894, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • J.A. Delgado-Guerrero, A. Colomé and C. Torras. Contextual policy search for micro-data robot motion learning through covariate Gaussian process latent variable models, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020, Las Vegas, NV, USA, pp. 5511-5517.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, J. Segovia, C. Torras and G. Alenyà. STRIPS action discovery, 2020 AAAI 2020 workshop on Generalization in Planning, 2020, New York, pp. 1-9.

    Open/Close abstract Abstract Info Info pdf PDF

  • J.A. Delgado-Guerrero, A. Colomé and C. Torras. Sample-efficient robot motion learning using Gaussian process latent variable models, 2020 IEEE International Conference on Robotics and Automation, 2020, Paris, France, pp. 314-320, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Sanchez and F. Moreno-Noguer. Integrating human body mocaps into Blender using RGB images, 13th International Conference on Advances in Computer-Human Interactions, 2020, Valencia, pp. 285-290.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, P. Radeva and M. Dimiccoli. Activities of daily living monitoring via a wearable camera: toward real-world applications. IEEE Access, 8: 77344-77363, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Donaire, J. Borràs, G. Alenyà and C. Torras. A versatile gripper for cloth manipulation. IEEE Robotics and Automation Letters, 5(4): 6520-6527, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is the main facility devoted to experimentation on the topics of the research line. It occupies 142 m² in the second floor of the Faculty, and a large part hosts a life-scale mock-up of a (quite) small apartment (35 m²). Two PAL one-armed mobile TIAGo robots dwell within the apartment, but are occasionally taken out for experimentation elsewhere. There is also a manipulation area equipped with two WAM robot arms and a KINOVA manipulator, and workplaces are distributed along the perimeter. Next to the laboratory, with a direct view through a window, there is the Scientific and Technical Lab support office.

The Laboratory is further equipped with commercial and self-developed grippers, sensing devices, and augmented reality appliances. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosted the Humanoids Lab Initiative in the past, and we still keep 15 small humanoid robots for educational or promotional purposes.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

TFG Students

Support Staff