Research Project

RobInstruct: Instructing robots using natural communication skills

Type

National Project

Start Date

01/01/2015

End Date

31/12/2017

Project Code

TIN2014-58178-R

Project illustration

Staff

Project Description

The demographic projections for the EU (low birth rate and an ageing population) open a series of social, medical and economical challenges that demand for new technologies for the assistance of elderly and disabled people, both indoors and outdoors. A key aspect which should be researched is that of devising such new user-centric technologies in a way that they can easily adapt to the specific needs of the end-users.

In robotics, the goal of allowing an inexperienced end-user to program a robot with a new desired behavior has been pursued using Learning from Demonstration approaches, where the human teaches the robot by simply showing it how to perform the task. A typical set-up consists of a manipulator arm, teleoperated by the user through a haptic device to, for instance, pour water into a glass. The demonstration is usually repeated several times so the robot can learn the different steps, the essence, and the variability of the task. Yet, this procedure is not as natural as we would expect. Ideally, we would like to get rid of any specialized device and teach the robot as we teach another person, just by showing how the human performs the task or simply narrating the steps needed. The robot should be able to perceive the meaningful actions, identify the tools being used, and extract the essential knowledge of the process to be able to actually perform the entire task itself. The first main objective of the RobInstruct project is thus to move a step forward from existing approaches in robot learning, and develop the technology to instruct a general-purpose robot in a natural and human-like manner.

Additionally, most general-purpose robotic assistants designed so far, are restricted to indoor and non-urban scenarios, ignoring the important part of the therapy that represents the social interaction out-of-home. As a second objective of the project we will therefore pursue a number of assistance tasks in urban areas, such as streets, a university campus or a shopping mall.

To tackle these problems, we will put together tools from computer vision, machine learning, natural language processing and robotics. Specifically, to teach robots in a natural and human-like manner we will first develop parsers to represent both video and natural language data using an intermediate abstraction level. We will then investigate learning approaches to discover mappings between the visual/textual content and the robot action space. These algorithms should address several issues: heterogeneity of possible end- users, differences between the human and the robot kinematics, and large amounts of perceptual noise (as in outdoor settings), an element typically overlooked in the past.

In order to bring these assistive robots to outdoor and uncontrolled scenarios, the human-to- robot communication skills will be combined with new algorithms to reliably localize the robot in very large 3D maps and for long periods of time, even in GPS-denied areas. For this purpose, we will integrate novel computer vision pose estimation algorithms with inertial sensors.

The two main objectives we pursue are commercially and socially relevant robotics technologies, as endorsed by our three EPOs. With the development of such technologies, the project will contribute to more flexible and general use of robots in assistive tasks, as well as more robust service robots able to navigate and interact in previously unknown and unstructured environments.

Project Publications

Journal Publications

  • E. Corona, G. Alenyà, A. Gabás and C. Torras. Active garment recognition and target grasping point detection using deep learning. Pattern Recognition, 74: 629-641, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Santamaria-Navarro, G. Loianno, J. Solà, V. Kumar and J. Andrade-Cetto. Autonomous navigation of micro aerial vehicles: State estimation using fast and low-cost sensors. Autonomous Robots, 42(6): 1263-1280, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Jevtić, A. Colomé, G. Alenyà and C. Torras. Robot motion adaptation through user intervention and reinforcement learning. Pattern Recognition Letters, 105: 67-75, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. A scalable, efficient, and accurate solution to non-rigid structure from motion. Computer Vision and Image Understanding, 167: 121-133, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, F. Yan, F. Moreno-Noguer and K. Mikolajczyk. BreakingNews: Article annotation by image and text processing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): 1072-1085, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Vallvé, J. Solà and J. Andrade-Cetto. Graph SLAM sparsification with populated topologies using factor descent optimization. IEEE Robotics and Automation Letters, 3(2): 1322-1329, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Force-based representation for non-rigid shape and elastic model estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(9): 2137-2150, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • M. Villamizar, J. Andrade-Cetto, A. Sanfeliu and F. Moreno-Noguer. Boosted random ferns for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(2): 272-288, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Santamaria-Navarro, P. Grosch, V. Lippiello, J. Solà and J. Andrade-Cetto. Uncalibrated visual servo for unmanned aerial manipulation. IEEE/ASME Transactions on Mechatronics, 22(4): 1610-1621, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Combining local-physical and global-statistical models for sequential deformable shape from motion. International Journal of Computer Vision, 122(2): 371-387, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • P. Jiménez. Visual grasp point localization, classification and state recognition in robotic manipulation of cloth: An overview. Robotics and Autonomous Systems, 92: 107-125, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • R. Rossi, A. Santamaria-Navarro, J. Andrade-Cetto and P. Rocco. Trajectory generation for unmanned aerial manipulators through quadratic programming. IEEE Robotics and Automation Letters, 2(2): 389-396, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Porzi, S. Rota-Bulò, A. Penate-Sanchez, E. Ricci and F. Moreno-Noguer. Learning depth-aware deep representations for robotic perception. IEEE Robotics and Automation Letters, 2(2): 468-475, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, H. Schulz, B. Dellen, C. Torras and S. Behnke. Combining semantic and geometric features for object class segmentation of indoor scenes. IEEE Robotics and Automation Letters, 2(1): 49-55, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, J.M. Martínez, L. Agapito and B. Calvo. Modal space: A physics-based model for sequential estimation of time-varying shape from monocular video. Journal of Mathematical Imaging and Vision, 57(1): 75–98, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. 3D human pose tracking priors using geodesic mixture models. International Journal of Computer Vision, 122(2): 388-408, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Deray, J. Solà and J. Andrade-Cetto. Word ordering and document adjacency for large loop closure detection in 2D laser maps. IEEE Robotics and Automation Letters, 2(3): 1532-1539, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà, T. Ribeiro, K. Inoue and C. Torras. Relational reinforcement learning for planning with exogenous effects. Journal of Machine Learning Research, 18(78): 1-44, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Dual REPS: A generalization of relative entropy policy search exploiting bad experiences. IEEE Transactions on Robotics, 33(4): 978-985, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Moreno-Noguer and J.M. Porta. A Bayesian approach to simultaneously recover camera pose and non-rigid shape from monocular images. Image and Vision Computing, 52: 141-153, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Sanromà, A. Penate-Sanchez, R. Alquézar Mancho, F. Serratosa, F. Moreno-Noguer, J. Andrade-Cetto and M.A. González. MSClique: Multiple structure discovery through the maximum weighted clique problem. PLOS One, 11(1): e0145846, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • V. Lippiello, J. Cacace, A. Santamaria-Navarro, J. Andrade-Cetto, M.A. Trujillo, Y. Rodriguez and A. Viguria. Hybrid visual servoing with hierarchical task composition for aerial manipulation. IEEE Robotics and Automation Letters, 1(1): 259-266, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. Service robots for citizens of the future. European Review, 24(1): 17-30, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Execution Fault Recovery in Robot Programming by Demonstration Using Multiple Models. IEEE Latin America Transactions , 14(2): 517-523, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Real-time 3D reconstruction of non-rigid shapes with a single moving camera. Computer Vision and Image Understanding, 153(12): 37–54, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Incremental learning of skills in a task-parameterized Gaussian mixture model. Journal of Intelligent and Robotic Systems, 82(1): 81-99, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Sequential non-rigid structure from motion using physical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5): 979-994, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Action recognition based on efficient deep feature learning in the spatio-temporal domain. IEEE Robotics and Automation Letters, 1(2): 984-991, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. A 3D descriptor to detect task-oriented grasping points in clothing. Pattern Recognition, 60: 936-948, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Interactive multiple object learning with scanty human supervision. Computer Vision and Image Understanding, 149: 51-64, 2016.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • A. Agudo and F. Moreno-Noguer. DUST: Dual union of spatio-temporal subspaces for monocular multiple object 3D reconstruction, 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, HW, USA, pp. 1513-1521, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Hernandez Ruiz, L. Porzi, S. Rota and F. Moreno-Noguer. 3D CNNs on distance matrices for human action recognition, 25th ACM Conference on Multimedia, 2017, Mountain View, CA. USA., pp. 1087-1095.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Lunni, A. Santamaria-Navarro, R. Rossi, P. Rocco, L. Bascetta and J. Andrade-Cetto. Nonlinear model predictive control for aerial manipulation, 2017 International Conference on Unmanned Aircraft Systems, 2017, Miami, pp. 87-93.

    Open/Close abstract Abstract Info Info pdf PDF
  • V. Vaquero, G. Ros, F. Moreno-Noguer, A. M. and A. Sanfeliu. Joint coarse-and-fine reasoning for deep optical flow, 24th IEEE International Conference on Image Processing, 2017, Beijing, pp. 2558-2562.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Global model with local interpretation for dynamic shape reconstruction, 17th IEEE Winter Conference on Applications of Computer Vision, 2017, Santa Rosa, CA, USA, pp. 264-272, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. Multi-modal joint embedding for fashion product retrieval, 24th IEEE International Conference on Image Processing, 2017, Beijing, pp. 400-404.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Moreno-Noguer. 3D human pose estimation from a single image via distance matrix regression, 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, HW, USA, pp. 1561-1570, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Porzi, A. Penate-Sanchez, E. Ricci and F. Moreno-Noguer. Depth-aware convolutional neural networks for accurate 3D pose estimation in RGB-D images, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017, Vancouver, Canada, pp. 5777-5783.

    Open/Close abstract Abstract Info Info pdf PDF
  • V. Vaquero, I. del Pino, F. Moreno-Noguer, J. Solà, A. Sanfeliu and J. Andrade-Cetto. Deconvolutional networks for point-cloud vehicle detection and tracking in driving scenarios, 8th European Conference on Mobile Robots, 2017, Paris, France, pp. 1-7.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Canal, G. Alenyà and C. Torras. A taxonomy of preferences for physically assistive robots, 26th IEEE International Symposium on Robot and Human Interactive Communication, 2017, Lisbon, Portugal, pp. 292-297.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa. Multimodal news article analysis, 26th International Joint Conference on Artificial Intelligence, 2017, Melbourne, Australia, pp. 5136-5140.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Vallvé, J. Solà and J. Andrade-Cetto. Factor descent optimization for sparsification in graph SLAM, 8th European Conference on Mobile Robots, 2017, Paris, France, pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF
  • P. Jiménez. Software tools for the cognitive development of autonomous robots, 2017 International Workshop on Sense and Scope of Autonomy in Emerging Military and Security Technologies, 2017, Barcelona, pp. 18-32, The Richardson Institute, Lancaster University, UK.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Pumarola, A. Vakhitov, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. PL-SLAM: Real-time monocular visual SLAM with points and lines, 2017 IEEE International Conference on Robotics and Automation, 2017, Singapore, pp. 4503-4508.

    Open/Close abstract Abstract Info Info pdf PDF
  • I. del Pino, V. Vaquero, B. Masini, J. Solà, F. Moreno-Noguer, A. Sanfeliu and J. Andrade-Cetto. Low resolution lidar-based multi object tracking for driving applications, 3rd Iberian Robotics Conference, 2017, Seville, in Robot 2017: Third Iberian Robotics Conference, Vol 694 of Advances in Intelligent Systems and Computing, pp. 287-298, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • N. Covallero, D. Martínez, G. Alenyà and C. Torras. Planning clearing actions in cluttered scenes by phasing in geometrical constraints, 20th Catalan Conference on Artificial Intelligence, 2017, Deltebre, in Recent Advances in Artificial Intelligence Research and Development, Vol 300 of Frontiers in Artificial Intelligence and Applications, pp. 233-238, 2017, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Recovering pose and 3D deformable shape from multi-instance image ensembles, 13th Asian Conference on Computer Vision, 2016, Taipei, Taiwan, in Computer Vision – ACCV 2016, Vol 10114 of Lecture Notes in Computer Science, pp. 291-307, 2017, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Canal, G. Alenyà and C. Torras. Personalization framework for adaptive robotic feeding assistance, 8th International Conference on Social Robotics, 2016, Kansas City, USA, in Social Robotics, Vol 9979 of Lecture Notes in Computer Science, pp. 22-31, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Jevtić, A. Colomé, G. Alenyà and C. Torras. User evaluation of an interactive learning framework for single-arm and dual-arm robots, 8th International Conference on Social Robotics, 2016, Kansas City, USA, in Social Robotics, Vol 9979 of Lecture Notes in Computer Science, pp. 52-61, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, J.M. Martínez, B. Calvo and F. Moreno-Noguer. Mode-shape interpretation: Re-thinking modal space for recovering deformable shapes, 16th IEEE Winter Conference on Applications of Computer Vision, 2016, Lake Placid, NY, USA, pp. 1-8, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà, C. Torras, T. Ribeiro and K. Inoue. Learning relational dynamics of stochastic domains for planning, 26th International Conference on Automated Planning and Scheduling, 2016, London, pp. 235-243.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Martín, F. Husain, H. Schulz, S. Frintrop, C. Torras and S. Behnke. Semantic segmentation priors for object discovery, 23rd International Conference on Pattern Recognition, 2016, Cancún, Mexico, pp. 549-554.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. BASS: Boundary-aware superpixel segmentation, 23rd International Conference on Pattern Recognition, 2016, Cancún, Mexico, pp. 2824-2829.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Vakhitov, J. Funke and F. Moreno-Noguer. Accurate and linear time pose estimation from points and lines, 14th European Conference on Computer Vision, 2016, Amsterdam, Netherlands, in Computer Vision - ECCV 2016, Vol 9911 of Lecture Notes in Computer Science, pp. 583-599, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Quattoni, A. Ramisa, P. Swaroop, E. Simo-Serra and F. Moreno-Noguer. Structured prediction with output embeddings for semantic image annotation, 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2016, San Diego, pp. 552-557.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Corominas Murtra, J. Vallvé, J. Solà, I. Flores and J. Andrade-Cetto. Observability analysis and optimal sensor placement in stereo radar odometry, 2016 IEEE International Conference on Robotics and Automation, 2016, Stockholm, Sweden, pp. 3161-3166.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Gabás, E. Corona, G. Alenyà and C. Torras. Robot-aided cloth classification using depth information and CNNs, 9th Conference on Articulated Motion and Deformable Objects, 2016, Palma de Mallorca, Spain, in Articulated Motion of Deformable Objects, Vol 9756 of Lecture Notes in Computer Science, pp. 16-23, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Penate-Sanchez, L. Porzi and F. Moreno-Noguer. Matchability prediction for full-search template matching algorithms, 2015 International Conference on 3D Vision, 2015, Lyon, pp. 353-361.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Learning shape, motion and elastic models in force space, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 756-764, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • L.D. Ellebracht, A. Ramisa, P. Swaroop, J.A. Cordero, F. Moreno-Noguer and A. Quattoni. Semantic tuples for evaluation of image sentence generation, 4th Workshop on Vision and Language, 2015, Lisbon.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, J. Wang, Y. Lu, E. Dellandrea, F. Moreno-Noguer and R. Gaizauskas. Combining geometric, textual and visual features for predicting prepositions in image descriptions, 10th Conference on Empirical Methods in Natural Language Processing, 2015, Lisbon, pp. 214-220.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, E. Trulls Fortuny, L. Ferraz, I. Kokkinos, P. Fua and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 118-126, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

Other Publications

  • A. Colomé and C. Torras. Reinforcement Learning of Bimanual Robot Skills. Volume 134 of Springer Tracts in Advanced Robotics. Springer, 2020.

    Open/Close abstract Abstract Info Info
  • A. Santamaria-Navarro, J. Solà and J. Andrade-Cetto. Visual Guidance of Unmanned Aerial Manipulators. Volume 125 of Springer Tracts in Advanced Robotics. Springer International Publishing, 2019.

    Open/Close abstract Abstract Info Info
  • R. Valencia and J. Andrade-Cetto. Mapping, Planning and Exploration with Pose SLAM. Volume 119 of Springer Tracts in Advanced Robotics. Springer, 2018.

    Open/Close abstract Abstract Info Info
  • F. Husain, B. Dellen and C. Torras. Scene understanding using deep learning. In Handbook of Neural Computation, 373-382. Elsevier, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. Robot pain: A speculative review of its functions. In Pain and the Conscious Brain, 235-246. Wolters Kluwer, 2016.

    Open/Close abstract Abstract Info Info pdf PDF