Research line

Perception and Manipulation Image

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.

Head of line: Carme Torras Genís

Head of line

Tech. transfer

Our activity finds applications in several fields through collaboration with our technological partners

Research projects

We carry out projects from national and international research programmes.
→ More about our research projects

<< Back to Perception and Manipulation main page

Learning by demonstration

We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

→ more info

Research area 1 of Perception and Manipulation

Planning for perception and manipulation

We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

→ more info

Research area 2 of Perception and Manipulation

Perception of rigid and non-rigid objects

Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

→ more info

Research area 3 of Perception and Manipulation

These are the latest research projects of the Perception and Manipulation research line:

These are the most recent publications (2016 - 2015) of the Perception and Manipulation

  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Execution Fault Recovery in Robot Programming by Demonstration Using Multiple Models. IEEE Latin America Transactions , 14(2): 517-523, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Real-time 3D reconstruction of non-rigid shapes with a single moving camera. Computer Vision and Image Understanding, 2016, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Relational reinforcement learning with guided demonstrations. Artificial Intelligence, 2016, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • N. Covallero and G. Alenyà. Grasping novel objects. Technical Report IRI-TR-16-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agostini, C. Torras and F. Wörgötter. Efficient interactive decision-making framework for robotic applications. Artificial Intelligence, 2016, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Incremental learning of skills in a task-parameterized Gaussian mixture model. Journal of Intelligent and Robotic Systems, 82(1): 81-99, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Sequential non-rigid structure from motion using physical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5): 979-994, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Trulls Fortuny, I. Kokkinos, A. Sanfeliu and F. Moreno-Noguer. Dense segmentation-aware descriptors. In Dense Image Correspondences for Computer Vision, 83-107. Springer, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Sanromà, A. Penate-Sanchez, R. Alquézar Mancho, F. Serratosa, F. Moreno-Noguer, J. Andrade-Cetto and M.A. González. MSClique: Multiple structure discovery through the maximum weighted clique problem. PLOS One, 11(1): e0145846, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. Service robots for citizens of the future. European Review, 24(1): 17-30, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, J.M. Martínez, B. Calvo and F. Moreno-Noguer. Mode-shape interpretation: Re-thinking modal space for recovering deformable shapes, 2016 IEEE Winter Conference on Applications of Computer Vision, 2016, Lake Placid (USA), IEEE, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà, C. Torras, T. Ribeiro and K. Inoue. Learning relational dynamics of stochastic domains for planning, 2016 International Conference on Automated Planning and Scheduling, 2016, London, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Canal, S. Escalera and C. Angulo. A real-time human-robot interaction system based on gestures for assistive scenarios. Computer Vision and Image Understanding, 2016, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Modeling robot's world with minimal effort, 2015 IEEE International Conference on Robotics and Automation, 2015, Seattle, pp. 4890-4896.

    Open/Close abstract Abstract Info Info pdf PDF
  • B. Dellen, H. Scharr and C. Torras. Growth signature of rosette plants from time-lapse video. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 12(6): 1470-1478, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Multimodal object recognition using random clustering trees, 7th Iberian Conference on Pattern Recognition and Image Analysis, 2015, Santiago de Compostela, in Pattern Recognition and Image Analysis, Vol 9117 of Lecture Notes in Computer Science, pp. 496-504, 2015, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • L.D. Ellebracht, A. Ramisa, P. Swaroop, J.A. Cordero, F. Moreno-Noguer and A. Quattoni. Semantic tuples for evaluation of image sentence generation, 4th Workshop on Vision and Language, 2015, Lisbon.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Angulo, S. Pfeiffer, R. Tellez and G. Alenyà. Evaluating the use of robots to enlarge AAL services. Journal of Ambient Intelligence and Smart Environments, 7(3): 301-313, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, J. Wang, Y. Lu, E. Dellandrea, F. Moreno-Noguer and R. Gaizauskas. Combining geometric, textual and visual features for predicting prepositions in image descriptions, 2015 Conference on Empirical Methods in Natural Language Processing, 2015, Lisbon, pp. 214-220.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Marquès and G. Alenyà. Planning trajectories for the Esautomatix workcell. Technical Report IRI-TR-15-06, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, E. Trulls Fortuny, L. Ferraz, I. Kokkinos, P. Fua and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 118-126, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, S. Fidler, F. Moreno-Noguer and R. Urtasun. Neuroaesthetics in fashion: Modeling the perception of fashionability, 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, Boston.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Penate-Sanchez, L. Porzi and F. Moreno-Noguer. Matchability prediction for full-search template matching algorithms, 2015 International Conference on 3D Vision, 2015, Lyon, pp. 353-361.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. Lie algebra-based kinematic prior for 3D human pose tracking, 14th IAPR International Conference on Machine Vision Applications, 2015, Tokyo, Japan, pp. 394-397.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. DaLI: Deformation and light invariant descriptor. International Journal of Computer Vision, 115(2): 136–154, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Alexandre, J. Cristophe, V. Ruiz de Angulo and C. Torras. Variable symmetry breaking in numerical constraint problems. Artificial Intelligence, 229: 105-125, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Closed-loop inverse kinematics for redundant robots: Comparative assessment and two enhancements. IEEE/ASME Transactions on Mechatronics, 20(2): 944-955, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Safe robot execution in model-based reinforcement learning, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015, Hamburg, pp. 6422-6427.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Consistent depth video segmentation using adaptive surface models. IEEE Transactions on Cybernetics, 45(2): 266-278, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, T. Ribeiro, K. Inoue, G. Alenyà and C. Torras. Learning probabilistic action models from interpretation transitions, 31st Technical Communications of the International Conference on Logic Programming, 2015, Cork, Ireland, in Technical Communications of ICLP 2015, Vol 1433 of CEUR Workshop Proceedings.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Serradell, M.A. Pinheiro, R. Sznitman, J. Kybic, F. Moreno-Noguer and P. Fua. Non-rigid graph registration using active testing search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3): 625-638, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • S. Foix, G. Alenyà and C. Torras. 3D sensor planning framework for leaf probing, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015, Hamburg, pp. 6501-6506.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Planning robot manipulation to clean planar surfaces. Engineering Applications of Artificial Intelligence, 39: 23-32, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Alenyà, S. Foix and C. Torras. ToF cameras for eye-in-hand robotics. In Optical Imaging Devices: New Technologies and Applications, 117-147. CRC Press - Francis Taylor Group, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. V-MIN: Efficient reinforcement learning through demonstrations and relaxed reward demands, 29th AAAI Conference on Artificial Intelligence, 2015, Austin, Texas, pp. 2857-2863.

    Open/Close abstract Abstract Info Info pdf PDF
  • Y. Isleyici and G. Alenyà. Teaching grasping points using natural movements, 18th Catalan Conference on Artificial Intelligence, 2015, Valencia, in Artificial Intelligence Research and Development, Vol 277 of Frontiers in Artificial Intelligence and Applications, pp. 275-278, 2015, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • S. Hernández and A. Gabás. Staübli work-cell: General description and operation. Technical Report IRI-TR-15-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Simultaneous pose and non-rigid shape with particle dynamics, 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, Boston, pp. 2179-2187.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé, A. Planells and C. Torras. A friction-model-based framework for reinforcement learning of robotic tasks in non-rigid environments, 2015 IEEE International Conference on Robotics and Automation, 2015, Seattle, pp. 5649-5654.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Learning shape, motion and elastic models in force space, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 756-764, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Rubio, M. Villamizar, L. Ferraz, A. Penate-Sanchez, A. Ramisa, E. Simo-Serra, A. Sanfeliu and F. Moreno-Noguer. Efficient monocular pose estimation for complex 3D models, 2015 IEEE International Conference on Robotics and Automation, 2015, Seattle, pp. 1397-1402.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Jevtic, G. Doisy, Y. Parmet and Y. Edan. Comparison of interaction modalities for mobile indoor robot guidance: Direct physical interaction, person following, and pointing control. IEEE Transactions on Human-Machine Systems, 45(6): 653-663, 2015.

    Open/Close abstract Abstract Info Info pdf PDF

Perception and Manipulation Laboratory

The Perception and Manipulation Laboratory is equipped with 2 workcells: one with 2 standard manipulators and an XY positioner, and the other with 2 WAM arms in a reconfigurable arrangement. Additionally, researchers can find a 3 fingered hand, a Delta haptic interface, force sensors, several conventional cameras, and high speed, high resolution, and 3D cameras. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosts the Humanoid Lab initiative, with 15 small humanoid robots.

Perception and Manipulation Laboratory
Group photo

Researchers

PhD Students

Master Students

PFC Students