Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2020 - 2019) of the Perception and Manipulation

  • A. Cartas, P. Radeva and M. Dimiccoli. Activities of daily living monitoring via a wearable camera: toward real-world applications. IEEE Access, 8: 77344-77363, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs, G. Alenyà and C. Torras. A grasping-centered analysis for cloth manipulation. IEEE Transactions on Robotics, 36(3): 924-936, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Jangir, G. Alenyà and C. Torras. Dynamic cloth manipulation with deep reinforcement learning, 2020 IEEE International Conference on Robotics and Automation, 2020, Online, pp. 4630-4636, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà and F. Moreno-Noguer. Context-aware human motion prediction, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, pp. 6992-7001, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, A. Pumarola, G. Alenyà, F. Moreno-Noguer and G. Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, pp. 5031-5041, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, S. Popov, F. Moreno-Noguer and V. Ferrari. C-Flow: Conditional generative flow models for images and 3D point clouds, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Segmentation and 3D reconstruction of non-rigid shape from RGB video, 27th IEEE International Conference on Image Processing, 2020, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot. In Fault Diagnosis and Fault-tolerant Control of Robotic Systems, . The Institution of Engineering and Technology, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Segovia, S. Jiménez and A. Jonsson. Generalized planning with positive and negative examples, 34th AAAI Conference on Artificial Intelligence, 2020, New York, USA, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez and C. Torras. Perception of cloth in assistive robotic manipulation tasks. Natural Computing, 19(2): 409-431, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Short-term human-robot interaction adaptability in real-world environments. International Journal of Social Robotics, 12: 639-657, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Vendrell, C. Sun, V. Puig and G. Cembrano. Fuzzy clustering application on failure rate prediction in water distribution networks. Technical Report IRI-TR-20-02, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128: 698-713, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Dual-branch CNNs for vehicle detection and tracking on LiDAR data. IEEE Transactions on Intelligent Transportation Systems, 2020, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, M. Lippi, M.C. Welle, H. Yin, R. Antanova, A. Varava, J. Borràs, C. Torras, A. Marino, G. Alenyà and D. Kragic. Benchmarking bimanual cloth manipulation. IEEE Robotics and Automation Letters, 5(2): 1111-1118, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Pignaton, J. Olszewska, J.L. Carbonera, S. Fiorini, A. Khamis, V.R. Kumar, M. Barreto, E. Prestes, M.K. Habib, S. Redfield, A. Chibani, P. Goncalves, J. Bermejo-Alonso, R. Sanz, E. Tosello, A. Olivares-Alarcos, A.A. Konzen, J. Quintas and H. Li. Ontological concepts for information sharing in cloud robotics. Journal of Ambient Intelligence and Humanized Computing, 2020, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Reinforcement Learning of Bimanual Robot Skills. Volume 134 of Springer Tracts in Advanced Robotics. Springer, 2020.

    Open/Close abstract Abstract Info Info

  • V. Sidhu, E. Tretschk, V. Golyanik, A. Agudo and C. Theobalt. Neural dense non-rigid structure from motion with latent space constraints, 16th European Conference on Computer Vision, 2020, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Mitjans, M. Maceira and G. Alenyà. Corner detection of deformable fabric using deep learning. Technical Report IRI-TR-20-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • J.C. Gassó, A. Olivares-Alarcos and G. Alenyà. Visual feedback for humans about robots' perception in collaborative environments. Technical Report IRI-TR-20-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Cognitive system framework for brain-training exercise based on human-robot interaction. Cognitive Computation, 12: 793-810, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, R. Huertas, S. Forgas, C. Torras and G. Alenyà. Discovering SOCIABLE: Using a conceptual model to evaluate the legibility and effectiveness of backchannel cues in an entertainment scenario, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Parent, A. Colomé and C. Torras. Variable impedance control in Cartesian latent space while avoiding obstacles in null space, 2020 IEEE International Conference on Robotics and Automation, 2020, Online, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Science-fiction: A mirror for the future of humankind. IDEES(48): 1-11, 2020.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Shape basis interpretation for monocular deformable 3-D reconstruction. IEEE Transactions on Multimedia, 21(4): 821-834, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Learning robot policies using a high-level abstraction persona-behaviour simulator, 28th IEEE International Symposium on Robot and Human Interactive Communication, 2019, New Delhi, India, pp. 1-8.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Sanchez, P. Radeva and M. Dimiccoli. Social relation recognition in photostreams, 26th IEEE International Conference on Image Processing, 2019, Taipei, Taiwan, pp. 3227-3231.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Amadio, A. Colomé and C. Torras. Exploiting symmetries in reinforcement learning of bimanual robotic tasks. IEEE Robotics and Automation Letters, 4(2): 1838-1845, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, A. Suárez, J. Segovia, C. Torras and G. Alenyà. Natural teaching of robot-assisted rearranging exercises for cognitive training, 11th International Conference on Social Robotics, 2019, Madrid, in Social Robotics, Vol 11876 of Lecture Notes in Computer Science, pp. 611-621, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Cartas, J. Luque, P. Radeva, C. Segura and M. Dimiccoli. Seeing and hearing egocentric actions: How much can we learn?, 2019 ICCV Workshop on Egocentric Perception, Interaction and Computing, 2019, Seoul, South Corea, pp. 4470-4480.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Simao, F. Martínez-Jerónimo, V. Blasco, F. Moreno-Noguer, J.M. Porta, J. Pestana, A. Soarez, D. Raldúa and C. Barata. Using a new high-throughput video-tracking platform to assess behavioural changes in Daphnia magna exposed to neuro-active drugs. Science of the Total Environment, 662: 160-167, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, K. Fischer, F. Moreno-Noguer, A. Sanfeliu and S. Milz. Improving map re-localization with deep 'movable' objects segmentation on 3D LiDAR point clouds, 2019 IEEE Intelligent Transportation Systems Conference, 2019, Auckland, New Zeland, pp. 942-949.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Semantic relatedness based re-ranker for text spotting, 2019 Conference on Empirical Methods in Natural Language Processing, 2019, Hong Kong.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, A. Sanfeliu and F. Moreno-Noguer. Online learning and detection of faces with low human supervision. The Visual Computer, 35(3): 349-370, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Garcia, A. Agudo and F. Moreno-Noguer. Vehicle pose estimation via regression of semantic points of interest, 11th IEEE International Symposium on Image and Signal Processing and Analysis, 2019, Dubrovnik, Croatia, pp. 6.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Belchí. Optimising the topological information of the A∞-persistence groups. Discrete & Computational Geometry, 62(1): 29-54, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Funke, F. Tschopp, W. Grisaitis, A. Sheridan, C. Singh, S. Saalfeld and S.C. Turaga. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7): 1669-1680, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, S. Foix and G. Alenyà. On inferring intentions in shared tasks for industrial collaborative robots. Electronics, 8(11): 1306, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Anticipatory science fiction to foster ethical debates on AI and robotics, 2019 International Symposium on Anticipation and Anticipatory Systems: Humans Meet AI, 2019, Örebro, Sweden, pp. 1-3.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, C. Torras and G. Alenyà. Practical resolution methods for MDPs in robotics exemplified with disassembly planning. IEEE Robotics and Automation Letters, 4(3): 2282-2288, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Garcia, F. Moreno-Noguer and A. Agudo. 3D vehicle detection on an FPGA from LiDAR point clouds, 2nd International Conference on Watermarking and Image Processing, 2019, Marseille, France.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dalmasso, A. Garrell Zulueta, P. Jiménez and A. Sanfeliu. Human-robot collaborative navigation search using social reward sources, 4th Iberian Robotics Conference, 2019, Porto, Portugal, Vol 1093 of Advances in Intelligent Systems and Computing, pp. 84-95, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Jangir, G. Alenyà and C. Torras. Learning cloth manipulation with demonstrations. Technical Report IRI-TR-19-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • V.A. Carmona, J. Lobo, J. van Ruysevelt, C. Torras and J.M. Font. Development and preliminary evaluation of the ArmTracker: a wearable system to monitor arm activity during daily life, IX Reunión del Capítulo Español de la Sociedad Europea de Biomecánica, 2019, Las Palmas de Gran Canaria.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez and G. Blanco. Effective computation of base points of ideals in two-dimensional local rings. Journal of Symbolic Computation, 92: 93-109, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Detailed 3D face reconstruction from a single RGB image, 27th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, 2019, Plzen, Czech Republic, Vol 27(2) of Journal of WSCG, pp. 103-112.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Rodríguez, J. Lobo and J.M. Font. A review of lower-limb wearable exoskeletons for overground rehabilitation, IX Reunión del Capítulo Español de la Sociedad Europea de Biomecánica, 2019, Las Palmas de Gran Canaria, pp. 23-24.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Single view facial hair 3D reconstruction, 2019 Iberian Conference on Pattern Recognition and Image Analysis, 2019, Madrid, Spain, pp. 423-436.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Robotics and artificial intelligence meet the humanities: Some initiatives for ethics education and dissemination. In Humanities and Higher Education: Synergies between Science, Technology and Humanities, 267-273. Global University Network for Innovation, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, M. Cashmore, S. Krivić, G. Alenyà, D. Magazzeni and C. Torras. Probabilistic planning for robotics with ROSPlan, 2019 Annual Conference Towards Autonomous Robotic Systems Conference, 2019, London, in Towards Autonomous Robotic Systems, Vol 11649 of Lecture Notes in Computer Science, pp. 236-250, 2019, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, J. Sanchez-Riera, G. P. T. Choi, A. Sanfeliu and F. Moreno-Noguer. 3DPeople: Modeling the geometry of dressed humans, 17th International Conference on Computer Vision, 2019, Seoul, pp. 2242-2251.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Assistive robotics: Research challenges and ethics education initiatives. DILEMATA: International Journal of Applied Ethics, 30: 63-77, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, V. Goswami, F. Vicente, F. de la Torre and F. Moreno-Noguer. Unsupervised image-to-video clothing transfer, 2019 ICCV Workshop on Computer Vision for Fashion, Art and Design, 2019, Seoul, pp. 3181-3184.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, J. Lobo, C. Torras and G. Alenyà. Robot interaction adaptation for healthcare assistance, 2019 Workshop on Artificial Intelligence & Robotics in Service Interactions, 2019, Zaragoza, pp. 159-172.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, D. Beßler, A. Khamis, P. Goncalves, M.K. Habib, J. Bermejo-Alonso, M. Barreto, M. Diab, J. Rosell, J. Quintas, J. Olszewska, H. Nakawala, E. Pignaton, A. Gyrard, S. Borgo, G. Alenyà, M. Beetz and H. Li. A review and comparison of ontology-based approaches to robot autonomy. The Knowledge Engineering Review, 34: e29, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Amor, A. Ruiz, F. Moreno-Noguer and A. Sanfeliu. Precise localization for aerial inspection using augmented reality markers. In Aerial Robotic Manipulation. Vol 129 of Springer Tracts in Advanced Robotics, 249-259. Springer, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • V.R. Kumar, A. Khamis, S. Fiorini, J.L. Carbonera, A. Olivares-Alarcos, M.K. Habib, P. Goncalves, H. Li and J. Olszewska. Ontologies for industry 4.0. The Knowledge Engineering Review, 34: e17, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Jevtić, A. Flores, G. Alenyà, G. Chance, P. Caleb-Solly, S. Dogramadzi and C. Torras. Personalized robot assistant for support in dressing. IEEE Transactions on Cognitive and Developmental Systems, 11(3): 363-374, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Vakhitov, A. Agudo, F. Moreno-Noguer and A. Sanfeliu. Relative localization for aerial manipulation with PL-SLAM. In Aerial Robotic Manipulation. Vol 129 of Springer Tracts in Advanced Robotics, 239-248. Springer, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Hernandez Ruiz, J. Gall and F. Moreno-Noguer. Human motion prediction via spatio-temporal inpainting, 17th International Conference on Computer Vision, 2019, Seoul, pp. 7133-7142.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, G. Alenyà and C. Torras. Adapting robot task planning to user preferences: An assistive shoe dressing example. Autonomous Robots, 43(6): 1343-1356, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot using a LPV robust unknown input observer, 4th Conference on Control and Fault Tolerant Systems, 2019, Casablanca, Morocco, pp. 207-212.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Lönn, P. Radeva and M. Dimiccoli. Smartphone picture organization: A hierarchical approach. Computer Vision and Image Understanding, 187: 102789:1-13, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4): 971-984, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • K. Bedin, S. Foix and G. Alenyà. Robots and IoT devices for assistive automation. Technical Report IRI-TR-19-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Dimiccoli and H. Wendt. Enhancing temporal segmentation by nonlocal self-similarity, 26th IEEE International Conference on Image Processing, 2019, Taipei, Taiwan, pp. 3681-3685.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is equipped with 2 workcells: one with 2 standard manipulators and an XY positioner, and the other with 2 WAM arms in a reconfigurable arrangement. Additionally, researchers can find a 3 fingered hand, a Delta haptic interface, force sensors, several conventional cameras, and high speed, high resolution, and 3D cameras. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosts the Humanoid Lab initiative, with 15 small humanoid robots.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

Support Staff