Research line
Perception and Manipulation

The research of PERCEPTION AND MANIPULATION group focuses on enhancing the perception, learning, and planning capabilities of robots to achieve higher degrees of autonomy and user-friendliness during everyday manipulation tasks. Some topics addressed are the geometric interpretation of perceptual information, construction of 3D object models, action selection and planning, reinforcement learning, and teaching by demonstration.
Head of line: Carme Torras Genís

Research areas
>> Learning by demonstration
>> Planning for perception and manipulation
>> Perception of rigid and non-rigid objects

Tech. transfer
Our activity finds applications in several fields through collaboration with our technological partners
Facilities

Research projects
We carry out projects from national and international research programmes.
→ More about our research projects
<< Back to Perception and Manipulation main page
Learning by demonstration
We devise methods to learn object-action relations to accomplish tasks at different levels of abstraction, where object models are generated from visual and depth information, and actions, involving manipulation skills, are learned from demonstrations provided by a human using multimodal algorithms that combine vision and haptics.

Planning for perception and manipulation
We are interested in view planning for object modeling, as well as manipulation planning, with special interest in deformable objects. High-level task formulations are integrated with low-level geometry-based methods and simplified physical models, together with an on-line sensory-based treatment of uncertainty, so as to come up with specific sequences of motion commands.

Perception of rigid and non-rigid objects
Our objective is to investigate computer vision algorithms for interpreting and understanding scenes from images, with applications in robotics and medical imaging. In particular, our activities are concentrated on retrieving rigid and non-rigid shape, motion and camera pose from single images and video sequences.

These are the latest research projects of the Perception and Manipulation research line:
-
IPALM: Interactive Perception-Action-Learning for Modelling Objects
European Project
Start Date: 01/10/2019
-
IMAGINE: Robots understanding their actions by imagining their effects
European Project
Start Date: 01/01/2017
-
I-DRESS: Assistive interactive robotic system for support in dressing
European Project
Start Date: 01/12/2015
-
ViSen: Visual Sense, Tagging visual data with semantic descriptions
European Project
Start Date: 01/07/2013
-
IntellAct: Intelligent observation and execution of Actions and manipulations
European Project
Start Date: 01/03/2011
-
PACOPLUS: Perception, action & cognition through learning of object-action complexes.
European Project
Start Date: 01/02/2006
-
PARALIN: Parallel computing modelling for industrial problems
European Project
Start Date: 01/06/1996
-
GREAT: Beyond Graph Neural Networks: Joint graph topology learning and graph-based inference for computer vision
National Project
Start Date: 01/06/2020
-
HuMoUR: Markerless 3D human motion understanding for adaptive robot behavior
National Project
Start Date: 01/01/2018
-
RobInstruct: Instructing robots using natural communication skills
National Project
Start Date: 01/01/2015
-
REST: Computación mediante restricciones en robótica y gestión de recursos
National Project
Start Date: 31/12/1999
-
MARE: Manipulación robotizada: Aprendizaje por demostración y Refuerzo
CSIC Project
Start Date: 15/12/2018
-
R3OBJ: Reconstrucción 3D, localización y segmentación automática de objetos a partir de imágenes
CSIC Project
Start Date: 22/11/2018
-
CINNOVA: Modelos cinemáticos y técnicas de aprendizaje para robots de estructura innovadora
CSIC Project
Start Date: 01/07/2011
-
MVOD: Medición eficiente de volúmenes de objetos deformables con cámaras rentables
CSIC Project
Start Date: 01/03/2012
-
APREN: Modelos perceptivos y técnicas de aprendizaje para robots de servicios
CSIC Project
Start Date: 01/01/2010
-
BPnP: Priors para la detección 3D en tiempo real de objetos rígidos y deformables
CSIC Project
Start Date: 01/10/2008
-
SGR RobIRI: Grup consolidat de Percepció i Manipulació Robotitzada de l'IRI
Regional Project
Start Date: 01/01/2017
-
SIMBIOTS: Facilitar una introducció de la robòtica a nous processos i aplicacions dintre de la industria
Regional Project
Start Date: 01/01/2018
-
PYERCING: Open your ears: predicting object material from impact sounds
UPC Project
Start Date: 01/07/2020
-
MP-CloL: Learning Robotic Cloth Manipulation based on Physics Models and Model Predictive Control
UPC Project
Start Date: 01/07/2020
-
SPECTACLE: From Spectators to Actors: Understanding Social Interactions through the Eyes of a Participant
UPC Project
Start Date: 01/10/2019
-
MESSI: Camera Self-Calibration for Structure from Motion on Dynamic Scenarios
UPC Project
Start Date: 01/10/2019
-
iMOVE: Development of a hybrid control interface based on sEMG and movement signals for an assistive robotic manipulator
UPC Project
Start Date: 01/10/2019
-
LatCom: Variable Impedance Control with Movement Primitives in Latent Spaces During Compliant Manipulation
UPC Project
Start Date: 01/01/2019
-
DeepSoccer: Deep Learning Technology applied to Soccer Videos
Technology Transfer Contract
Start Date: 20/05/2019
-
SLD: Proceso automatizado de detección y clasificación inteligente de pieles (smart leather detector)
Technology Transfer Contract
Start Date: 06/03/2018
-
AIPICS: Desarrollo de un sistema integral de control de la producción dotado de inteligencia artificial y capacidad de autodecisión
Technology Transfer Contract
Start Date: 01/01/2018
-
RELAPP: Aplicació electrònica portàtil controlada per una APP per a la mesura in situ de l’elasticitat de materials visco-elàstics
Technology Transfer Contract
Start Date: 19/10/2020
-
SCIACE: Sistema de climatización inteligente auto-adaptativo a las condiciones del entorno
Technology Transfer Contract
Start Date: 15/12/2017
-
Google ResearchAward: GANimation3D: Unsupervised 3D Face Animation from Monocular Images
Technology Transfer Contract
Start Date: 01/10/2019
-
Amazon ResearchAward: Geometry-aware 3D Human Body Animation from Still Photos
Technology Transfer Contract
Start Date: 14/03/2019
-
3T-SLTE: Grupo tractor para elevadores con gestión inteligente de energía integrada
Technology Transfer Contract
Start Date: 11/03/2015
-
PERSEO: Paquete escalable robusto soluciones específicas orientadas
Technology Transfer Contract
Start Date: 26/05/2016
-
Google Faculty Award: Geometry-aware CNNs for Non-Rigid Shape Reconstruction
Technology Transfer Contract
Start Date: 17/02/2017
-
TERMOSOLD: Desarrollo de un sistema basado en técnicas de inteligencia artificial para la inspección de soldaduras en lineas de envasado horizontal
Technology Transfer Contract
Start Date: 10/03/2015
-
CRISTALBREAD: Investigació de la capacitat de tecnologíes de robotització intel.ligent per a la manipulació de masses de panificació altament hidratades
Technology Transfer Contract
Start Date: 01/03/2015
-
SENECA: Semi-automatic neuron reconstruction in CATMAID
Technology Transfer Contract
Start Date: 01/09/2015
-
SITVIA: Sistema de inspección de tampografía por visión artificial adaptable
Technology Transfer Contract
Start Date: 07/09/2015
-
AutomaticTV: Algorithms for automatic audiovisual production
Technology Transfer Contract
Start Date: 24/10/2014
-
RecycLearn: Desarrollo de tecnología para aprendizaje autónomo de un brazo robótico - Aplicación a plantas de reciclaje.
Technology Transfer Contract
Start Date: 01/09/2013
-
ROBOSOFT: Sistema de visió d'ajuda a la conducció per al vehicle NICOLAS-VTCU
Technology Transfer Contract
Start Date: 23/02/2006
These are the most recent publications (2021 - 2020) of the Perception and Manipulation
-
J. Garcia, A. Agudo and F. Moreno-Noguer. E-DNAS: Differentiable neural architecture search for embedded systems, 25th International Conference on Pattern Recognition, 2021, Online, to appear.
Abstract
Info
PDF
-
A. Agudo. Total estimation from RGB video: On-line camera self-calibration, non-rigid shape and motion, 25th International Conference on Pattern Recognition, 2021, Online, to appear.
Abstract
Info
PDF
-
A. Andriella, C. Torras and G. Alenyà. Short-term human-robot interaction adaptability in real-world environments. International Journal of Social Robotics, 12: 639-657, 2020.
Abstract
Info
PDF
-
P. Jiménez and C. Torras. Perception of cloth in assistive robotic manipulation tasks. Natural Computing, 19(2): 409-431, 2020.
Abstract
Info
PDF
-
M. Dimiccoli and H. Wendt. Learning event representations for temporal segmentation of image sequences by dynamic graph embedding. IEEE Transactions on Image Processing, 30: 1476-1486, 2020.
Abstract
Info
PDF
-
A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu and F. Moreno-Noguer. GANimation: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128: 698-713, 2020.
Abstract
Info
PDF
-
J. Vendrell, C. Sun, V. Puig and G. Cembrano. Fuzzy clustering application on failure rate prediction in water distribution networks. Technical Report IRI-TR-20-02, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.
Abstract
Info
PDF
-
I. Garcia-Camacho, M. Lippi, M.C. Welle, H. Yin, R. Antanova, A. Varava, J. Borràs, C. Torras, A. Marino, G. Alenyà and D. Kragic. Benchmarking bimanual cloth manipulation. IEEE Robotics and Automation Letters, 5(2): 1111-1118, 2020.
Abstract
Info
PDF
-
V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Dual-branch CNNs for vehicle detection and tracking on LiDAR data. IEEE Transactions on Intelligent Transportation Systems, 2020, to appear.
Abstract
Info
PDF
-
A. Colomé and C. Torras. Reinforcement Learning of Bimanual Robot Skills. Volume 134 of Springer Tracts in Advanced Robotics. Springer, 2020.
Abstract
Info
-
E. Pignaton, J. Olszewska, J.L. Carbonera, S. Fiorini, A. Khamis, V.R. Kumar, M. Barreto, E. Prestes, M.K. Habib, S. Redfield, A. Chibani, P. Goncalves, J. Bermejo-Alonso, R. Sanz, E. Tosello, A. Olivares-Alarcos, A.A. Konzen, J. Quintas and H. Li. Ontological concepts for information sharing in cloud robotics. Journal of Ambient Intelligence and Humanized Computing, 2020, to appear.
Abstract
Info
PDF
-
A. Mitjans, M. Maceira and G. Alenyà. Corner detection of deformable fabric using deep learning. Technical Report IRI-TR-20-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.
Abstract
Info
PDF
-
V. Sidhu, E. Tretschk, V. Golyanik, A. Agudo and C. Theobalt. Neural dense non-rigid structure from motion with latent space constraints, 16th European Conference on Computer Vision, 2020, Online, in Computer Vision - ECCV 2020, Vol 12361 of Lecture Notes in Computer Science, pp. 204-222, 2020.
Abstract
Info
PDF
-
A. Andriella, C. Torras and G. Alenyà. Cognitive system framework for brain-training exercise based on human-robot interaction. Cognitive Computation, 12: 793-810, 2020.
Abstract
Info
PDF
-
J.C. Gassó Loncan Vallecillo, A. Olivares-Alarcos and G. Alenyà. Visual feedback for humans about robots' perception in collaborative environments. Technical Report IRI-TR-20-03, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.
Abstract
Info
PDF
-
D. Parent, A. Colomé and C. Torras. Variable impedance control in Cartesian latent space while avoiding obstacles in null space, 2020 IEEE International Conference on Robotics and Automation, 2020, Online, to appear.
Abstract
Info
PDF
-
A. Andriella, R. Huertas, S. Forgas, C. Torras and G. Alenyà. Discovering SOCIABLE: Using a conceptual model to evaluate the legibility and effectiveness of backchannel cues in an entertainment scenario, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Online, pp. 752-759.
Abstract
Info
PDF
-
J.A. Delgado-Guerrero, A. Colomé and C. Torras. Sample-efficient robot motion learning using Gaussian process latent variable models, 2020 IEEE International Conference on Robotics and Automation, 2020, Online, to appear.
Abstract
Info
PDF
-
C. Torras. Science-fiction: A mirror for the future of humankind. IDEES(48): 1-11, 2020.
Abstract
Info
PDF
-
A. Cartas, P. Radeva and M. Dimiccoli. Activities of daily living monitoring via a wearable camera: toward real-world applications. IEEE Access, 8: 77344-77363, 2020.
Abstract
Info
PDF
-
J.A. Delgado-Guerrero, A. Colomé and C. Torras. Contextual policy search for micro-data robot motion learning through covariate Gaussian process latent variable models, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020, Las Vegas, NV, USA, to appear.
Abstract
Info
PDF
-
J. Borràs, G. Alenyà and C. Torras. A grasping-centered analysis for cloth manipulation. IEEE Transactions on Robotics, 36(3): 924-936, 2020.
Abstract
Info
PDF
-
J. Sanchez-Riera and F. Moreno-Noguer. Integrating human body mocaps into Blender using RGB images, 13th International Conference on Advances in Computer-Human Interactions, 2020, Valencia, pp. 285-290.
Abstract
Info
PDF
-
R. Jangir, G. Alenyà and C. Torras. Dynamic cloth manipulation with deep reinforcement learning, 2020 IEEE International Conference on Robotics and Automation, 2020, Online, pp. 4630-4636.
Abstract
Info
PDF
-
S. Donaire, J. Borràs, G. Alenyà and C. Torras. A versatile gripper for cloth manipulation. IEEE Robotics and Automation Letters, 5(4): 6520-6527, 2020.
Abstract
Info
PDF
-
E. Corona, A. Pumarola, G. Alenyà and F. Moreno-Noguer. Context-aware human motion prediction, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, pp. 6990-6999.
Abstract
Info
PDF
-
M. Maceira, A. Olivares-Alarcos and G. Alenyà. Recurrent neural networks for inferring intentions in shared tasks for industrial collaborative robots, 29th IEEE International Symposium on Robot and Human Interactive Communication, 2020, Online, pp. 665-670.
Abstract
Info
PDF
-
E. Corona, A. Pumarola, G. Alenyà, F. Moreno-Noguer and G. Rogez. GanHand: Predicting human grasp affordances in multi-object scenes, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, pp. 5030-5040.
Abstract
Info
PDF
-
F. Coltraro. Experimental validation of an inextensible cloth model. Technical Report IRI-TR-20-04, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2020.
Abstract
Info
PDF
-
A. Pumarola, S. Popov, F. Moreno-Noguer and V. Ferrari. C-Flow: Conditional generative flow models for images and 3D point clouds, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Online, to appear.
Abstract
Info
PDF
-
F. Strazzeri and C. Torras. Configuration space of a textile rectangle. In Geometry of Varieties and Applications, . Springer-Birkhäuser, 2020.
Abstract
Info
PDF
-
A. San Miguel, V. Puig and G. Alenyà. Fault-tolerant control of a service robot. In Fault Diagnosis and Fault-tolerant Control of Robotic Systems, . The Institution of Engineering and Technology, 2020.
Abstract
Info
PDF
-
A. Agudo. Segmentation and 3D reconstruction of non-rigid shape from RGB video, 27th IEEE International Conference on Image Processing, 2020, Online, pp. 2845-2849.
Abstract
Info
PDF
-
A. Andriella, H. Siqueira, D. Fu, S. Magg, P. Barros, S. Wermter, C. Torras and G. Alenyà. Do I have a personality? Endowing care robots with context-dependent personality traits. International Journal of Social Robotics, 2020, to appear.
Abstract
Info
PDF
-
J. Segovia, S. Jiménez and A. Jonsson. Generalized planning with positive and negative examples, 34th AAAI Conference on Artificial Intelligence, 2020, New York, USA, to appear.
Abstract
Info
PDF
-
A. Agudo. Unsupervised 3D reconstruction and grouping of rigid and non-rigid categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, to appear.
Abstract
Info
PDF
-
I. Geer, M. Maceira, J. Borràs, C. Torras and G. Alenyà. Interaction identification through tactile sensing during cloth manipulation using a 3-axis touch sensor, 2020 IROS Workshop on Robotic Manipulation of Deformable Objects, 2020, Online, pp. 1-6.
Abstract
Info
PDF
Perception and Manipulation Laboratory
The Perception and Manipulation Laboratory is the main facility devoted to experimentation on the topics of the research line. It occupies 142 m² in the second floor of the Faculty, and a large part hosts a life-scale mock-up of a (quite) small apartment (35 m²). Two PAL one-armed mobile TIAGo robots dwell within the apartment, but are occasionally taken out for experimentation elsewhere. There is also a manipulation area equipped with two WAM robot arms and a KINOVA manipulator, and workplaces are distributed along the perimeter. Next to the laboratory, with a direct view through a window, there is the Scientific and Technical Lab support office.
The Laboratory is further equipped with commercial and self-developed grippers, sensing devices, and augmented reality appliances. Laboratory service offers quick experimental setup, several standardized software tools, and expertise in robot control and perception algorithms. It also hosted the Humanoids Lab Initiative in the past, and we still keep 15 small humanoid robots for educational or promotional purposes.


Researchers
-
Agudo Martínez, Antonio
aagudo (at) iri.upc.edu
-
Alberich Carramiñana, Maria
alberich (at) iri.upc.edu
-
Alenyà Ribas, Guillem
galenya (at) iri.upc.edu
-
Borràs Sol, Júlia
jborras (at) iri.upc.edu
-
Colomé Figueras, Adrià
acolome (at) iri.upc.edu
-
Dimiccoli, Mariella
mdimiccoli (at) iri.upc.edu
-
Grosch Obregon, Patrick John
pgrosch (at) iri.upc.edu
-
Jiménez Schlegl, Pablo
jimenez (at) iri.upc.edu
-
Moreno Noguer, Francesc
fmoreno (at) iri.upc.edu
-
Ruiz Ovejero, Adrià
aruiz (at) iri.upc.edu
-
Sanchez-Riera, Jordi
jsanchez (at) iri.upc.edu
-
Strazzeri, Fabio
fstrazzeri (at) iri.upc.edu
-
Torras Genís, Carme
torras (at) iri.upc.edu
PhD Students
-
Amadio, Fabio
famadio (at) iri.upc.edu
-
Andriella, Antonio
aandriella (at) iri.upc.edu
-
Coltraro Ianniello, Franco
fcoltraro (at) iri.upc.edu
-
Corona Puyané, Enric
ecorona (at) iri.upc.edu
-
Delgado Guerrero, Juan Antonio
jdelgado (at) iri.upc.edu
-
Garcia López, Javier
jgarcia (at) iri.upc.edu
-
Garcia-Camacho, Irene
igarcia (at) iri.upc.edu
-
Gonzalez Fernandez, Antonio
agonzalezf (at) iri.upc.edu
-
Hernández Ruiz, Alejandro José
ahernandez (at) iri.upc.edu
-
Izquierdo Badiola, Silvia
sizquierdo (at) iri.upc.edu
-
Olivares-Alarcos, Alberto
aolivares (at) iri.upc.edu
-
Pareto Boada, Júlia
jpareto (at) iri.upc.edu
-
Ponsa Cobas, Sergi
sponsa (at) iri.upc.edu
-
Reixach Pérez, David
dreixach (at) iri.upc.edu
-
Rubio Romano, Antonio
arubio (at) iri.upc.edu
-
Shen, Jianxiong
jshen (at) iri.upc.edu
-
Suárez Hernández, Alejandro
asuarez (at) iri.upc.edu
-
Tzelepis, Georgios
gtzelepis (at) iri.upc.edu
-
Ugrinovic, Nicolás
nugrinovic (at) iri.upc.edu
Master Students
-
Arduengo García, Miguel
marduengo (at) iri.upc.edu
-
Dhamnaskar, Ameya
adhamanaskar (at) iri.upc.edu
-
Furlan, Serena
sfurlan (at) iri.upc.edu
-
Miranda Moreno, Antonio
amiranda (at) iri.upc.edu
-
Omodolor, Stevedan Ogochukwu
somodolor (at) iri.upc.edu
-
Ordoñez Apraez, Daniel Felipe
dordonez (at) iri.upc.edu
-
Parent Alonso, David
dparent (at) iri.upc.edu
-
Rana, Aditya Sangram Singh
arana (at) iri.upc.edu
-
Retyk, Federico
fretyk (at) iri.upc.edu
-
Tura Vecino, Biel
btura (at) iri.upc.edu
-
Vendrell Gallart, Joan
jvendrell (at) iri.upc.edu
Support Staff
-
Civit Bertran, Aniol
acivit (at) iri.upc.edu
-
Martínez Sánchez, Sergi
smartinezs (at) iri.upc.edu
-
Riba Pi, Edgar
eriba (at) iri.upc.edu
Follow us!