Research Project
IntellAct: Intelligent observation and execution of Actions and manipulations
Type
European Project
Start Date
01/03/2011
End Date
30/05/2015
Project Code
FP7-ICT-2009-6-269959
Staff
-
-
Thomas, Federico
Researcher
-
Ruiz de Angulo, Vicente
Researcher
-
Foix, Sergi
Researcher
-
Jiménez, Pablo
Researcher
-
Rozo, Leonel
Researcher
-
Pardo, Diego Esteban
Researcher
-
Alenyà, Guillem
Researcher
-
Andrade, Juan
Researcher
-
Dellen, Babette
Researcher
-
Agostini, Alejandro Gabriel
Researcher
-
Moreno, Francesc
Member
-
Martínez, David
Member
Project Description
IntellAct addresses the problem of understanding and exploiting the meaning (semantics) of manipulations in terms of objects, actions and their consequences for reproducing human actions with machines. It comprises three building blocks: (1) Learning: Abstract, semantic descriptions of manipulations are extracted from video sequences showing a human demonstrating the manipulations; (2) Monitoring: In the second step, observed manipulations are evaluated against the learned, semantic models; (3) Execution: Based on learned, semantic models, equivalent manipulations are executed by a robot. The analysis of low-level observation data for semantic content (Learning) and the synthesis of concrete behavior (Execution) constitute the major scientific challenge of IntellAct. Based on the semantic interpretation and description and enhanced with low-level trajectory data for grounding, two major application areas are addressed by IntellAct: First, the monitoring of human manipulations for correctness (e.g., for training or in high-risk scenarios) and second, the efficient teaching of cognitive robots for manipulation (e.g. in the context of assembly processes in industrial settings and on a space station). To achieve these goals, IntellAct brings together recent methods for (1) parsing scenes into spatio-temporal graphs and so called ‘semantic event chains’, (2) probabilistic models of objects and their manipulation, (3) probabilistic rule learning, and (4) dynamic motion primitives for trainable and flexible descriptions of robotic motor behavior. Its implementation employs a concurrent-engineering approach that includes virtual-reality-enhanced simulation as well as physical robots. Its goal culminates in the demonstration of a robot understanding, monitoring and reproducing human action.
Project Publications
Journal Publications
-
S. Thiusius, B. Anders, S. Christian, N. Wantia, J. Rossmann, D. Martínez, G. Alenyà, C. Torras, U. Ales, N. Bojan, K. Aljaž, F. Wörgötter, E.E. Aksoy, J. Papon, S. Haller, J. Piater and N. Krüger. Teaching a robot the semantics of assembly tasks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(5): 670-692, 2018.
Abstract Info PDF
-
D. Martínez, G. Alenyà and C. Torras. Relational reinforcement learning with guided demonstrations. Artificial Intelligence, 247: 295–312, 2017.
Abstract Info PDF
-
C. Torras. Service robots for citizens of the future. European Review, 24(1): 17-30, 2016.
Abstract Info PDF
-
F. Husain, B. Dellen and C. Torras. Consistent depth video segmentation using adaptive surface models. IEEE Transactions on Cybernetics, 45(2): 266-278, 2015.
Abstract Info PDF
-
D. Martínez, G. Alenyà and C. Torras. Planning robot manipulation to clean planar surfaces. Engineering Applications of Artificial Intelligence, 39: 23-32, 2015.
Abstract Info PDF
-
E. Simo-Serra, C. Torras and F. Moreno-Noguer. DaLI: Deformation and light invariant descriptor. International Journal of Computer Vision, 115(2): 136–154, 2015.
Abstract Info PDF
-
A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. Learning RGB-D descriptors of garment parts for informed robot grasping. Engineering Applications of Artificial Intelligence, 35: 246-258, 2014.
Abstract Info PDF
-
F. Husain, B. Dellen and C. Torras. Robust surface tracking in range image sequences. Digital Signal Processing, 35: 37-44, 2014.
Abstract Info PDF
-
L. Rozo, P. Jiménez and C. Torras. A robot learning from demonstration framework to perform force-based manipulation tasks. Intelligent Service Robotics, 6(1): 33-51, 2013.
Abstract Info PDF
-
P. Jiménez. Survey on assembly sequencing: A combinatorial and geometrical perspective. Journal of Intelligent Manufacturing, 24(2): 235-250, 2013.
Abstract Info PDF
-
A. Penate-Sanchez, J. Andrade-Cetto and F. Moreno-Noguer. Exhaustive linearization for robust camera pose and focal length estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(10): 2387-2400, 2013.
Abstract Info PDF
-
A. Abramov, K. Pauwels, J. Papon, F. Wörgötter and B. Dellen. Real-time segmentation of stereo videos on a portable system with a mobile GPU. IEEE Transactions on Circuits and Systems for Video Technology, 22(9): 1292-1305, 2012.
Abstract Info PDF
-
S. Ulbrich, V. Ruiz de Angulo, C. Torras, T. Asfour and R. Dillman. Kinematic Bézier maps. IEEE Transactions on Systems, Man and Cybernetics: Part B, 42(4): 1215-1230, 2012.
Abstract Info PDF
Conference Publications
-
F. Husain, A. Colomé, B. Dellen, G. Alenyà and C. Torras. Realtime tracking and grasping of a moving object from range video, 2014 IEEE International Conference on Robotics and Automation, 2014, Hong Kong, China, pp. 2617-2622.
Abstract Info PDF
-
D. Martínez, G. Alenyà, P. Jiménez, C. Torras, J. Rossmann, N. Wantia, A. Eren Erdal, S. Haller and J. Piater. Active learning of manipulation sequences, 2014 IEEE International Conference on Robotics and Automation, 2014, Hong Kong, China, pp. 5671-5678.
Abstract Info PDF
-
A. Colomé and C. Torras. Dimensionality reduction and motion coordination in learning trajectories with dynamic movement primitives, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, Chicago, IL, USA, pp. 1414-1420.
Abstract Info PDF
-
D. Martínez, G. Alenyà and C. Torras. Finding safe policies in model-based active learning, 2014 IROS Machine Learning in Planning and Control of Robot Motion Workshop, 2014, Chicago, pp. 1-6.
Abstract Info PDF
-
F. Husain, B. Dellen and C. Torras. Recognizing point clouds using conditional random fields, 22nd International Conference on Pattern Recognition, 2014, Stockhom, Sweden, pp. 4257-4262, IEEE Computer Society.
Abstract Info PDF
-
T.R. Savarimuthu, A.G. Buch, Y. Yang, W. Mustafar, S. Haller, J. Papon, D. Martínez and E.E. Aksoy. Manipulation monitoring and robot intervention in complex manipulation sequences, 2014 RSS Workshop on Robotic Monitoring, 2014, Berkeley, CA, USA.
Abstract Info PDF
-
B. Dellen, F. Husain and C. Torras. Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations, 8th International Conference on Computer Vision Theory and Applications, 2013, Barcelona, pp. 244-251.
Abstract Info PDF
-
E. Simo-Serra, A. Quattoni, C. Torras and F. Moreno-Noguer. A joint model for 2D and 3D pose estimation from a single image, 2013 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2013, Portland, OR, USA, pp. 3634--3641.
Abstract Info PDF
-
A. Colomé, D. Pardo, G. Alenyà and C. Torras. External force estimation during compliant robot manipulation, 2013 IEEE International Conference on Robotics and Automation, 2013, Karlsruhe, Germany, pp. 3535-3540.
Abstract Info PDF
-
S. Reich, A. Abramov, J. Papon, F. Wörgötter and B. Dellen. A novel real-time edge-preserving smoothing filter, 8th International Conference on Computer Vision Theory and Applications, 2013, Barcelona, pp. 1-11.
Abstract Info PDF
-
A. Colomé, G. Alenyà and C. Torras. Handling high parameter dimensionality in reinforcement learning with dynamic motor primitives, 2013 ICRA Workshop on Novel Methods for Learning and Optimization of Control Policies and Trajectories for Robotics, 2013, Karlsruhe, Germany.
Abstract Info PDF
-
A. Colomé and C. Torras. Positioning two redundant arms for cooperative manipulation of objects, 6th International Workshop on Computational Kinematics, 2013, Barcelona, in Computational Kinematics, Vol 15 of Mechanisms and Machine Science, pp. 121-129, 2014, Springer.
Abstract Info PDF
-
D. Martínez, G. Alenyà and C. Torras. Planning surface cleaning tasks by learning uncertain drag actions outcomes, 2013 ICAPS Workshop on Planning and Robotics, 2013, Rome, pp. 106-111.
Abstract Info PDF
-
A. Ramisa and C. Torras. Large-scale image classification using ensembles of nested dichotomies, 16th Catalan Conference on Artificial Intelligence, 2013, Vic, Spain, in Artificial Intelligence Research and Development, Vol 256 of Frontiers in Artificial Intelligence and Applications, pp. 87-90, 2013, IOS Press.
Abstract Info PDF
-
A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. FINDDD: A fast 3D descriptor to characterize textiles for robot manipulation, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, Tokyo, Japan, pp. 824-830.
Abstract Info PDF
-
L. Rozo, S. Calinon, D. Caldwell, P. Jiménez and C. Torras. Learning collaborative impedance-based robot behaviors, 27th AAAI Conference on Artificial Intelligence, 2013, Bellevue, pp. 1422-1428.
Abstract Info PDF
-
L. Rozo, P. Jiménez and C. Torras. Force-based robot learning of pouring skills using parametric hidden Markov models, 9th International Workshop on Robot Motion and Control, 2013, Wasowo, Poland, pp. 227-232.
Abstract Info PDF
-
C. Torras. From the Turing test to science fiction: The challenges of social robotics, 16th Catalan Conference on Artificial Intelligence, 2013, Vic, Spain, in Artificial Intelligence Research and Development, Vol 256 of Frontiers in Artificial Intelligence and Applications, pp. 5-7, 2013, IOS Press.
Abstract Info PDF
-
G. Alenyà, A. Ramisa, F. Moreno-Noguer and C. Torras. Characterization of textile grasping experiments, 2012 ICRA Workshop on Conditions for Replicable Experiments and Performance Comparison in Robotics Research, 2012, St Paul, Minnesota, USA, pp. 1-6.
Abstract Info PDF
-
A. Colomé, D. Pardo, G. Alenyà and C. Torras. External force estimation for textile grasp detection, 2012 IROS Workshop Beyond Robot Grasping: Modern Approaches for Learning Dynamic Manipulation, 2012, Vilamoura, Portugal, pp. 1-1.
Abstract Info PDF
-
D. Pardo, L. Rozo, G. Alenyà and C. Torras. Dynamically consistent probabilistic model for robot motion learning, 2012 IROS Workshop on Learning and Interaction in Haptic Robots, 2012, Algarve, Portugal, pp. 1-2.
Abstract Info PDF
-
A. Alexey, K. Pauwels, J. Papon, F. Wörgötter and B. Dellen. Depth-supported real-time video segmentation with the Kinect, 2012 IEEE Workshop on Applications of Computer Vision, 2012, Breckenridge, CO, USA, pp. 457-464.
Abstract Info PDF
-
A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. Using depth and appearance features for informed robot grasping of highly wrinkled clothes, 2012 IEEE International Conference on Robotics and Automation, 2012, Saint Paul, MN, USA, pp. 1703-1708, IEEE.
Abstract Info PDF
-
S. Foix, S. Kriegel, S. Fuchs, G. Alenyà and C. Torras. Information-gain view planning for free-form object reconstruction with a 3D ToF camera, 14th International Conference on Advanced Concepts for Intelligent Vision Systems, 2012, Brno, Czech Republic, in Advanced Concepts for Intelligent Vision Systems, Vol 7517 of Lecture Notes in Computer Science, pp. 36-47, 2012, Springer.
Abstract Info PDF
-
F. Rigual, A. Ramisa, G. Alenyà and C. Torras. Object detection methods for robot grasping: Experimental assessment and tuning, 15th Catalan Conference on Artificial Intelligence, 2012, Alacant, in Artificial Intelligence Research and Development, Vol 248 of Frontiers in Artificial Intelligence and Applications, pp. 123-132, 2012, IOS Press.
Abstract Info PDF
-
E. Simo-Serra, A. Ramisa, G. Alenyà, C. Torras and F. Moreno-Noguer. Single image 3D human pose estimation from noisy observations, 2012 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2012, Providence, RI, USA, pp. 2673-2680, IEEE Press.
Abstract Info PDF
-
A. Colomé and C. Torras. Redundant inverse kinematics: Experimental comparative review and two enhancements, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Vilamoura, Portugal, pp. 5333-5340, IEEE.
Abstract Info PDF
-
P. Monsó, G. Alenyà and C. Torras. POMDP approach to robotized clothes separation, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Vilamoura, Portugal, pp. 1324-1329, IEEE.
Abstract Info PDF
-
L. Rozo, P. Jiménez and C. Torras. Robot learning from demonstration in the force domain, 2011 IJCAI Workshop on Agents Learning Interactively from Human Teachers, 2011, Barcelona, Spain, pp. 1-6.
Abstract Info PDF
-
L. Rozo, P. Jiménez and C. Torras. Robot learning from demonstration of force-based tasks with multiple solution trajectories, 15th International Conference on Advanced Robotics, 2011, Tallin, Estonia, pp. 124-129, IEEE.
Abstract Info PDF
-
E.E. Aksoy, B. Dellen, M. Tamosiunaite and F. Wörgötter. Execution of a dual-object (pushing) action with semantic event chains, 11th IEEE-RAS International Conference on Humanoid Robots, 2011, Bled, Slovenia, pp. 576-583, IEEE Press.
Abstract Info PDF
-
A.A. Ortega and J. Andrade-Cetto. Segmentation of dynamic objects from laser data, 5th European Conference on Mobile Robots, 2011, Örebro, Sweden, pp. 115-121, Örebro Univ.
Abstract Info PDF
-
E.H. Teniente and J. Andrade-Cetto. FaMSA: Fast multi-scan alignment with partially known correspondences, 5th European Conference on Mobile Robots, 2011, Örebro, Sweden, pp. 139-144, Örebro Univ.
Abstract Info PDF
-
E.H. Teniente, R. Valencia and J. Andrade-Cetto. Dense outdoor 3D mapping and navigation with Pose SLAM, 2011 Workshop de Robótica Experimental, 2011, Seville, pp. 567-572.
Abstract Info PDF
Follow us!