Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2019 - 2018) of the Perception and Manipulation

  • R. Jangir, G. Alenyà and C. Torras. Learning cloth manipulation with demonstrations. Technical Report IRI-TR-19-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Detailed 3D face reconstruction from a single RGB image, 27th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, 2019, Plzen, Czech Republic, Vol 27(2) of Journal of WSCG, pp. 103-112.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Moreno-Noguer, F. Lumbreras and A. Agudo. Single view facial hair 3D reconstruction, 2019 Iberian Conference on Pattern Recognition and Image Analysis, 2019, Madrid, Spain, pp. 423-436.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, M. Cashmore, S. Krivić, G. Alenyà, D. Magazzeni and C. Torras. Probabilistic planning for robotics with ROSPlan, 2019 Annual Conference Towards Autonomous Robotic Systems Conference, 2019, London, in Towards Autonomous Robotic Systems, Vol 11649 of Lecture Notes in Computer Science, pp. 236-250, 2019, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Assistive robotics: Research challenges and ethics education initiatives. DILEMATA: International Journal of Applied Ethics, 30: 63-77, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, J. Lobo, C. Torras and G. Alenyà. Robot interaction adaptation for healthcare assistance, 2019 Workshop on Artificial Intelligence & Robotics in Service Interactions, 2019, Zaragoza, pp. 159-172.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Amor, A. Ruiz, F. Moreno-Noguer and A. Sanfeliu. Precise localization for aerial inspection using augmented reality markers. In Aerial Robotic Manipulation. Vol 129 of Springer Tracts in Advanced Robotics, 249-259. Springer, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Vakhitov, A. Agudo, F. Moreno-Noguer and A. Sanfeliu. Relative localization for aerial manipulation with PL-SLAM. In Aerial Robotic Manipulation. Vol 129 of Springer Tracts in Advanced Robotics, 239-248. Springer, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Jevtić, A. Flores, G. Alenyà, G. Chance, P. Caleb-Solly, S. Dogramadzi and C. Torras. Personalized robot assistant for support in dressing. IEEE Transactions on Cognitive and Developmental Systems, 11(3): 363-374, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot using a LPV robust unknown input observer, 4th Conference on Control and Fault Tolerant Systems, 2019, Casablanca, Morocco, pp. 6, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, G. Alenyà and C. Torras. Adapting robot task planning to user preferences: An assistive shoe dressing example. Autonomous Robots, 43(6): 1343-1356, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot. In Fault Diagnosis and Fault-tolerant Control of Robotic Systems, . The Institution of Engineering and Technology, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4): 971-984, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • K. Bedin, S. Foix and G. Alenyà. Robots and IoT devices for assistive automation. Technical Report IRI-TR-19-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Shape basis interpretation for monocular deformable 3D reconstruction. IEEE Transactions on Multimedia, 21(4): 821-834, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, C. Torras and G. Alenyà. Learning robot policies using a high-level abstraction persona-behaviour simulator, 28th IEEE International Symposium on Robot and Human Interactive Communication, 2019, New Delhi, India, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Amadio, A. Colomé and C. Torras. Exploiting symmetries in reinforcement learning of bimanual robotic tasks. IEEE Robotics and Automation Letters, 4(2): 1838-1845, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, A. Suárez, J. Segovia, C. Torras and G. Alenyà. Natural teaching of robot-assisted rearranging exercises for cognitive training, 11th International Conference on Social Robotics, 2019, Madrid, in Social Robotics, Springer, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Simao, F. Martínez-Jerónimo, V. Blasco, F. Moreno-Noguer, J.M. Porta, J. Pestana, A. Soarez, D. Raldúa and C. Barata. Using a new high-throughput video-tracking platform to assess behavioural changes in Daphnia magna exposed to neuro-active drugs. Science of the Total Environment, 662: 160-167, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, K. Fischer, F. Moreno-Noguer, A. Sanfeliu and S. Milz. Improving map re-localization with deep 'movable' objects segmentation on 3D LiDAR point clouds, 2019 IEEE Intelligent Transportation Systems Conference, 2019, Auckland, New Zeland, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, A. Sanfeliu and F. Moreno-Noguer. Online learning and detection of faces with low human supervision. The Visual Computer, 35(3): 349-370, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, C. Torras and G. Alenyà. Practical resolution methods for MDPs in robotics exemplified with disassembly planning. IEEE Robotics and Automation Letters, 4(3): 2282-2288, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Force-based representation for non-rigid shape and elastic model estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(9): 2137-2150, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Deformable motion 3D reconstruction by union of regularized subspaces, 25th IEEE International Conference on Image Processing, 2018, Athens, Greece, pp. 2930-2934.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Visual re-ranking with natural language understanding for text spotting, 14th Asian Conference on Computer Vision, 2018, Perth, in Computer Vision – ACCV 2018, Vol 11363 of Lecture Notes in Computer Science, pp. 68-82, 2019, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, G. Alenyà, A. Gabás and C. Torras. Active garment recognition and target grasping point detection using deep learning. Pattern Recognition, 74: 629-641, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, E. Pignat, G. Alenyà, S. Calinon and C. Torras. Joining high-level symbolic planning with low-level motion primitives in adaptive HRI: application to dressing assistance, 2018 IEEE International Conference on Robotics and Automation, 2018, Brisbane, Australia, pp. 3273-3278.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Enhancing text spotting with a language model and visual context information, 21st Catalan Conference on Artificial Intelligence, 2018, Roses, in Artificial Intelligence Research and Development, Vol 308 of Frontiers in Artificial Intelligence and Applications, pp. 271-280, 2018, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. A scalable, efficient, and accurate solution to non-rigid structure from motion. Computer Vision and Image Understanding, 167: 121-133, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. The Vestigial Heart. A Novel of the Robot Age. The MIT Press, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Sabir, F. Moreno-Noguer and L. Padró. Visual semantic re-ranker for text spotting, 23rd Iberoamerican Congress on Pattern Recognition, 2018, Madrid, in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Vol 11401 of Lecture Notes in Computer Science, pp. 884-892, 2019.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes. IEEE Transactions on Robotics, 34(3): 602-615, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Dimensionality reduction in learning Gaussian mixture models of movement primitives for contextualized action selection and adaptation. IEEE Robotics and Automation Letters, 3(4): 3922-3929, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs Sol, R. Heudorfer, S. Rader, P. Kaiser and T. Asfour. The KIT swiss knife gripper for disassembly tasks: a multi-functional gripper for bimanual manipulation with a single arm, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, pp. 4590-4597.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez and F. Dachs. Constancy regions of mixed multiplier ideals in two-dimensional local rings with rational singularities. Mathematische Nachrichten, 291(2): 245-263, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Chance, A. Jevtić, P. Caleb-Solly, G. Alenyà, C. Torras and S. Dogramadzi. "Elbows out" - predictive tracking of partially occluded pose for robot-assisted dressing. IEEE Robotics and Automation Letters, 3(4): 3598-3605, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez and G. Čubrić. Robots in the textile and fashion industries: facts and prospectives, 2018 Scientific-Professional Symposium Textile Science and Economy, 2018, Zagreb, pp. 124-131.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ramisa, F. Yan, F. Moreno-Noguer and K. Mikolajczyk. BreakingNews: Article annotation by image and text processing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): 1072-1085, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: Anatomically-aware facial animation from a single image, 15th European Conference on Computer Vision, 2018, Munich, in Computer Vision - ECCV 2018, Vol 11214 of Lecture Notes in Computer Science, pp. 835-851, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez, F. Dachs and V. González-Alonso. Poincaré series for mixed multiplier ideals, XVI Encuentros de Álgebra Computacional y Aplicaciones, 2018, Zaragoza, in 43, pp. 27-30, Monografías de la Real Academia de Ciencias Exactas, Físicas, Químicas y Naturales de Zaragoza.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Foix, G. Alenyà and C. Torras. Task-driven active sensing framework applied to leaf probing. Computers and Electronics in Agriculture, 147: 166-175, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić and C. Torras. Adaptive modality selection algorithm in robot-assisted cognitive training, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, pp. 4456-4461.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez, F. Dachs and V. González-Alonso. Multiplicity and Poincaré series for mixed multiplier ideals, 5th Joint Meeting of the Czech, Slovenian, Austrian, Slovak and Catalan Mathematical Societies, 2018, Bratislava, Slovakia, pp. 59-61.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Tkach, A. Jevtić, S.Y. Nof and Y. Edan. A modified distributed bees algorithm for multi-sensor task allocation. Sensors, 18(3): 759:1-16, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, G. Alenyà and C. Torras. Interleaving hierarchical task planning and motion constraint testing for dual-arm manipulation, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, pp. 4061-4066.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Ramos, C.O. Scrob, A.S. Vázquez, R. Fernández and A. Olivares-Alarcos. Skill-oriented designer of conceptual robotic structures, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, pp. 5679-5684.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Deep lidar CNN to understand the dynamics of moving vehicles, 2018 IEEE International Conference on Robotics and Automation, 2018, Brisbane, Australia, pp. 4504-4509.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Hallucinating dense optical flow from sparse lidar for autonomous vehicles, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 1959-1964.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, M. Pijoan and F. Moreno-Noguer. Image collection pop-up: 3D reconstruction and clustering of rigid and non-rigid categories, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, pp. 2607-2615.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Garcia, A. Agudo and F. Moreno-Noguer. Vehicle pose estimation using G-Net: Multi-class localization and depth estimation, 21st Catalan Conference on Artificial Intelligence, 2018, Roses, in Artificial Intelligence Research and Development, Vol 308 of Frontiers in Artificial Intelligence and Applications, pp. 355-364, 2018, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit and F. Moreno-Noguer. Geometry-aware network for non-rigid shape prediction from a single view, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, pp. 4681-4690.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Aprenentatge Automàtic en Xarxes i Robots: Reptes Tecnocientífics i Implicacions Ètiques. Institut d'Estudis Catalans. Secció de Ciències i Tecnologia, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Unsupervised person image synthesis in arbitrary poses, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, pp. 8620-8628.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Social networks and robot companions: Technology, ethics and science fiction. Mètode Science Studies Journal, 99: 47-53, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić, J. Hernández-Farigola, N. Tantinyà, C. Abdelnour and C. Torras. Resource-based modality selection in robot-assisted cognitive training, 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, Chicago, USA, pp. 255-256.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras and R. López de Mántaras. Special issue editorial: Interconnectats: Màquines i humans davant del segle 10101. Mètode Science Studies Journal, 99: 2-3, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Thiusius, B. Anders, S. Christian, N. Wantia, J. Rossmann, D. Martínez, G. Alenyà, C. Torras, U. Ales, N. Bojan, K. Aljaž, F. Wörgötter, E.E. Aksoy, J. Papon, S. Haller, J. Piater and N. Krüger. Teaching a robot the semantics of assembly tasks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(5): 670-692, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, G. Alenyà, J. Hernández-Farigola and C. Torras. Deciding the different robot roles for patient cognitive training. International Journal of Human-Computer Studies, 117: 20-29, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Vila, G. Canal and G. Alenyà. Towards safety in physically assistive robots: eating assistance, 2018 IROS Workshop on Robots for Assisted Living, 2018, Madrid, pp. 1-4.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, J. Andrade-Cetto, A. Sanfeliu and F. Moreno-Noguer. Boosted random ferns for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(2): 272-288, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić and C. Torras. Adaptable multimodal interaction framework for robot-assisted cognitive training, 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, Chicago, USA, pp. 327-328.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Software tools for the cognitive development of autonomous robots. In Autonomy in Future Military and Security Technologies: Implications for Law, Peace, and Conflict, 18-32. Richardson Institute - Lancaster University , 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Jevtić, A. Colomé, G. Alenyà and C. Torras. Robot motion adaptation through user intervention and reinforcement learning. Pattern Recognition Letters, 105: 67-75, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Lumbreras, F. Moreno-Noguer and A. Agudo. 2D-to-3D facial expression transfer, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 2008-2013.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Pirashvili, L. Steinberg, F. Belchí, M. Niranjan, J.G. Frey and J. Brodzki. Improved understanding of aqueous solubility modeling through topological data analysis. Journal of Cheminformatics, 10(54): 1-14, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is equipped with 2 workcells: one with 2 standard manipulators and an XY positioner, and the other with 2 WAM arms in a reconfigurable arrangement. Additionally, researchers can find a 3 fingered hand, a Delta haptic interface, force sensors, several conventional cameras, and high speed, high resolution, and 3D cameras. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosts the Humanoid Lab initiative, with 15 small humanoid robots.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

TFG Students

Support Staff