Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2018 - 2017) of the Perception and Manipulation

  • A. Ramisa, F. Yan, F. Moreno-Noguer and K. Mikolajczyk. BreakingNews: Article annotation by image and text processing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): 1072-1085, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. The Vestigial Heart. A Novel of the Robot Age. The MIT Press, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Foix, G. Alenyà and C. Torras. Task-driven active sensing framework applied to leaf probing. Computers and Electronics in Agriculture, 147: 166-175, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Dimensionality reduction in learning Gaussian mixture models of movement primitives for contextualized action selection and adaptation. IEEE Robotics and Automation Letters, 3(4): 3922-3929, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Tkach, A. Jevtić, S.Y. Nof and Y. Edan. A modified distributed bees algorithm for multi-sensor task allocation. Sensors, 18(3): 759:1-16, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Chance, A. Jevtić, P. Caleb-Solly, G. Alenyà, C. Torras and S. Dogramadzi. "Elbows out" - predictive tracking of partially occluded pose for robot-assisted dressing. IEEE Robotics and Automation Letters, 3(4): 3598-3605, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Deep lidar CNN to understand the dynamics of moving vehicles, 2018 IEEE International Conference on Robotics and Automation, 2018, Brisbane, Australia, pp. 4504-4509.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: Anatomically-aware facial animation from a single image, 15th European Conference on Computer Vision, 2018, Munich, in Computer Vision - ECCV 2018, Vol 11214 of Lecture Notes in Computer Science, pp. 835-851, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Jevtić, A. Flores, G. Alenyà, G. Chance, P. Caleb-Solly, S. Dogramadzi and C. Torras. Personalized robot assistant for support in dressing. IEEE Transactions on Cognitive and Developmental Systems, 2018, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić and C. Torras. Adaptive modality selection algorithm in robot-assisted cognitive training, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, M. Pijoan and F. Moreno-Noguer. Image collection pop-up: 3D reconstruction and clustering of rigid and non-rigid categories, 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, G. Alenyà and C. Torras. Interleaving hierarchical task planning and motion constraint testing for dual-arm manipulation, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018, Madrid, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit and F. Moreno-Noguer. Geometry-aware network for non-rigid shape prediction from a single view, 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Shape basis interpretation for monocular deformable 3D reconstruction. IEEE Transactions on Multimedia, 2018, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pumarola, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Unsupervised person image synthesis in arbitrary poses, 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, A. Sanfeliu and F. Moreno-Noguer. Hallucinating dense optical flow from sparse lidar for autonomous vehicles, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 1959-1964.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Thiusius, B. Anders, S. Christian, N. Wantia, J. Rossmann, D. Martínez, G. Alenyà, C. Torras, U. Ales, N. Bojan, K. Aljaž, F. Wörgötter, E.E. Aksoy, J. Papon, S. Haller, J. Piater and N. Krüger. Teaching a robot the semantics of assembly tasks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(5): 670-692, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić, J. Hernández-Farigola, N. Tantinyà, C. Abdelnour and C. Torras. Resource-based modality selection in robot-assisted cognitive training, 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, Chicago, USA, pp. 255-256.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, J. Andrade-Cetto, A. Sanfeliu and F. Moreno-Noguer. Boosted random ferns for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(2): 272-288, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Andriella, G. Alenyà, J. Hernández-Farigola and C. Torras. Deciding the different robot roles for patient cognitive training. International Journal of Human-Computer Studies, 117: 20-29, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Jevtić, A. Colomé, G. Alenyà and C. Torras. Robot motion adaptation through user intervention and reinforcement learning. Pattern Recognition Letters, 105: 67-75, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Taranović, A. Jevtić and C. Torras. Adaptable multimodal interaction framework for robot-assisted cognitive training, 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, Chicago, USA, pp. 327-328.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Force-based representation for non-rigid shape and elastic model estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(9): 2137-2150, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, G. Alenyà and C. Torras. Adapting robot task planning to user preferences: An assistive shoe dressing example. Autonomous Robots, 2018, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, G. Alenyà, A. Gabás and C. Torras. Active garment recognition and target grasping point detection using deep learning. Pattern Recognition, 74: 629-641, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Rotger, F. Lumbreras, F. Moreno-Noguer and A. Agudo. 2D-to-3D facial expression transfer, 24th International Conference on Pattern Recognition, 2018, Beijing, China, pp. 2008-2013.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. A scalable, efficient, and accurate solution to non-rigid structure from motion. Computer Vision and Image Understanding, 167: 121-133, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Robust spatio-temporal clustering and reconstruction of multiple deformable bodies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes. IEEE Transactions on Robotics, 34(3): 602-615, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Deformable motion 3D reconstruction by union of regularized subspaces, 25th IEEE International Conference on Image Processing, 2018, Athens, Greece, pp. 2930-2934.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, J. Àlvarez and F. Dachs. Constancy regions of mixed multiplier ideals in two-dimensional local rings with rational singularities. Mathematische Nachrichten, 291(2): 245-263, 2018.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, E. Pignat, G. Alenyà, S. Calinon and C. Torras. Joining high-level symbolic planning with low-level motion primitives in adaptive HRI: application to dressing assistance, 2018 IEEE International Conference on Robotics and Automation, 2018, Brisbane, Australia, pp. 3273-3278.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Random clustering ferns for multimodal object recognition. Neural Computing and Applications, 28(9): 2445-2460, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, G. Ros, F. Moreno-Noguer, A. M. and A. Sanfeliu. Joint coarse-and-fine reasoning for deep optical flow, 24th IEEE International Conference on Image Processing, 2017, Beijing, pp. 2558-2562.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Els reptes de la robòtica assistencial. Memòries de la Reial Acadèmia de Ciències i Arts de Barcelona: 433-457/66/8, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Combining local-physical and global-statistical models for sequential deformable shape from motion. International Journal of Computer Vision, 122(2): 371-387, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. Multi-modal fashion product retrieval, 6th Workshop on Vision and Language, 2017, Valencia, pp. 43-45.

    Open/Close abstract Abstract Info Info pdf PDF

  • L. Porzi, S. Rota-Bulò, A. Penate-Sanchez, E. Ricci and F. Moreno-Noguer. Learning depth-aware deep representations for robotic perception. IEEE Robotics and Automation Letters, 2(2): 468-475, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. Global model with local interpretation for dynamic shape reconstruction, 2017 IEEE Winter Conference on Applications of Computer Vision, 2017, Santa Rosa (California), USA, pp. 264-272, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. Multi-modal joint embedding for fashion product retrieval, 24th IEEE International Conference on Image Processing, 2017, Beijing, pp. 400-404.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Funke, J. Klein, F. Moreno-Noguer, A. Cardona and M. Cook. TED: A tolerant edit distance for segmentation evaluation. Methods, 115: 119-127, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Bensch, A. Jevtić and T. Hellström. On interaction quality in human-robot interaction, 9th International Conference on Agents and Artificial Intelligence, 2017, Porto, Portugal, pp. 182-189.

    Open/Close abstract Abstract Info Info pdf PDF

  • V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Deconvolutional networks for point-cloud vehicle detection and tracking in driving scenarios, 8th European Conference on Mobile Robots, 2017, Paris, France, pp. 1-7.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Moreno-Noguer. 3D human pose estimation from a single image via distance matrix regression, 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, USA, pp. 1561-1570, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. 3D human pose tracking priors using geodesic mixture models. International Journal of Computer Vision, 122(2): 388-408, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Demonstration-free contextualized probabilistic movement primitives, further enhanced with obstacle avoidance, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017, Vancouver, Canada, pp. 3190-3195.

    Open/Close abstract Abstract Info Info pdf PDF

  • L. Porzi, A. Penate-Sanchez, E. Ricci and F. Moreno-Noguer. Depth-aware convolutional neural networks for accurate 3D pose estimation in RGB-D images, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017, Vancouver, Canada, pp. 5777-5783.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Garrell Zulueta, M. Villamizar, F. Moreno-Noguer and A. Sanfeliu. Teaching robot’s proactive behavior using human assistance. International Journal of Social Robotics, 9(2): 231-249, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Arduengo, S.J. Jorgensen, K. Hambuchen, L. Sentis, F. Moreno-Noguer and G. Alenyà. ROS wrapper for real-time multi-person pose estimation with a single camera. Technical Report IRI-TR-17-02, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ramisa, F. Yan, F. Moreno-Noguer and K. Mikolajczyk. The BreakingNews dataset, 6th Workshop on Vision and Language, 2017, Valencia, pp. 38-29.

    Open/Close abstract Abstract Info Info

  • A. Pumarola, A. Vakhitov, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. PL-SLAM: Real-time monocular visual SLAM with points and lines, 2017 IEEE International Conference on Robotics and Automation, 2017, Singapore, pp. 4503-4508.

    Open/Close abstract Abstract Info Info pdf PDF

  • C. Torras. Assistent(e)s robòtiqu(e)s: Un punt de confluència entre tecnociència i humanitats. Quadern de les idees, les arts i les lletres, 38(209): 19-21, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ramisa. Multimodal news article analysis, 26th International Joint Conference on Artificial Intelligence, 2017, Melbourne, Australia, pp. 5136-5140.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé and C. Torras. Dual REPS: A generalization of relative entropy policy search exploiting bad experiences. IEEE Transactions on Robotics, 33(4): 978-985, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, G. Alenyà and C. Torras. A taxonomy of preferences for physically assistive robots, 26th IEEE International Symposium on Robot and Human Interactive Communication, 2017, Lisbon, Portugal, pp. 292-297.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Software tools for the cognitive development of autonomous robots, 2017 International Workshop on Sense and Scope of Autonomy in Emerging Military and Security Technologies, 2017, Barcelona, pp. 18-32, The Richardson Institute, Lancaster University, UK.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Alberich-Carramiñana, B. Elizalde and F. Thomas. New algebraic conditions for the identification of the relative position of two coplanar ellipses. Computer Aided Geometric Design, 54: 35-48, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, C. Miralles and S. Medina. On the application of CMA-ES to biped walk. Technical Report IRI-TR-17-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo and F. Moreno-Noguer. DUST: Dual union of spatio-temporal subspaces for monocular multiple object 3D reconstruction, 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, USA, pp. 1513-1521, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Husain, B. Dellen and C. Torras. Scene understanding using deep learning. In Handbook of Neural Computation, 373-382. Elsevier, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agostini, G. Alenyà, A. Fischbach, H. Scharr, F. Wörgötter and C. Torras. A cognitive architecture for automatic gardening. Computers and Electronics in Agriculture, 138: 69-79, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. del Pino, V. Vaquero, B. Masini, J. Solà, F. Moreno-Noguer, A. Sanfeliu and J. Andrade-Cetto. Low resolution lidar-based multi object tracking for driving applications, 3rd Iberian Robotics Conference, 2017, Seville, in Robot 2017: Third Iberian Robotics Conference, Vol 694 of Advances in Intelligent Systems and Computing, pp. 287-298, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Lunni, A. Santamaria-Navarro, R. Rossi, P. Rocco, L. Bascetta and J. Andrade-Cetto. Nonlinear model predictive control for aerial manipulation, 2017 International Conference on Unmanned Aircraft Systems, 2017, Miami, pp. 87-93.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Martínez, G. Alenyà, T. Ribeiro, K. Inoue and C. Torras. Relational reinforcement learning for planning with exogenous effects. Journal of Machine Learning Research, 18(78): 1-44, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Martínez, G. Alenyà and C. Torras. Relational reinforcement learning with guided demonstrations. Artificial Intelligence, 247: 295–312, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Chance, A. Jevtić, P. Caleb-Solly and S. Dogramadzi. A quantitative analysis of dressing dynamics for robotic dressing assistance. Frontiers in Robotics and AI, 4(13): 1-14, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. Multi-modal embedding for main product detection in fashion, 2017 ICCV Workshop on Computer Vision for Fashion, 2017, Venice, Italy, pp. 2236-2242.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agostini, C. Torras and F. Wörgötter. Efficient interactive decision-making framework for robotic applications. Artificial Intelligence, 247: 187-212, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Chance, P. Caleb-Solly, A. Jevtić and S. Dogramadzi. What's “up”? — Resolving interaction ambiguity through non-visual cues for a robotic dressing assistant, 26th IEEE International Symposium on Robot and Human Interactive Communication, 2017, Lisbon, Portugal, pp. 284-291.

    Open/Close abstract Abstract Info Info pdf PDF

  • N. Covallero, D. Martínez, G. Alenyà and C. Torras. Planning clearing actions in cluttered scenes by phasing in geometrical constraints, 20th Catalan Conference on Artificial Intelligence, 2017, Deltebre, in Recent Advances in Artificial Intelligence Research and Development, Vol 300 of Frontiers in Artificial Intelligence and Applications, pp. 233-238, 2017, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Husain, H. Schulz, B. Dellen, C. Torras and S. Behnke. Combining semantic and geometric features for object class segmentation of indoor scenes. IEEE Robotics and Automation Letters, 2(1): 49-55, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Alenyà, R. Téllez, K. O'Regan and C. Angulo. Guest editorial sensorimotor contingencies for cognitive robotics. IEEE Transactions on Cognitive and Developmental Systems, 9(2): 97-99, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Colomé, S. Foix, G. Alenyà and C. Torras. Reward-weighted GMM and its application to action-selection in robotized shoe dressing, 3rd Iberian Robotics Conference, 2017, Seville, in Robot 2017: Third Iberian Robotics Conference, Vol 694 of Advances in Intelligent Systems and Computing, pp. 141-152, Springer.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo, J.M. Martínez, L. Agapito and B. Calvo. Modal space: A physics-based model for sequential estimation of time-varying shape from monocular video. Journal of Mathematical Imaging and Vision, 57(1): 75–98, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Visual grasp point localization, classification and state recognition in robotic manipulation of cloth: An overview. Robotics and Autonomous Systems, 92: 107-125, 2017.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Hernandez Ruiz, L. Porzi, S. Rota and F. Moreno-Noguer. 3D CNNs on distance matrices for human action recognition, 25th ACM Conference on Multimedia, 2017, Mountain View, CA. USA., pp. 1087-1095.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is equipped with 2 workcells: one with 2 standard manipulators and an XY positioner, and the other with 2 WAM arms in a reconfigurable arrangement. Additionally, researchers can find a 3 fingered hand, a Delta haptic interface, force sensors, several conventional cameras, and high speed, high resolution, and 3D cameras. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosts the Humanoid Lab initiative, with 15 small humanoid robots.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

TFG Students

Support Staff