Research Project

CHLOE-GRAPH: ClotH-Like ObjEcts Grasping, Representation, and Action Planning

Type

National Project

Start Date

01/09/2021

End Date

31/08/2024

Project Code

PID2020-118649RB-I00

Project illustration

Staff

Project Description

Project PID2020-118649RB-I00 funded by MCIN/ AEI /10.13039/501100011033

CHLOE-GRAPH proposes a novel robotic manipulation approach for cloth manipulation that is expected to be general enough to be applied to different assistive tasks. While rigid object manipulation has achieved a decent maturity, robots are yet to make an impact in the handling of deformable objects. Cloth-like objects are extremely challenging due to the complexity of understanding their deformation often with severe self-occlusions, making learning from real data very difficult. This has led to a tendency to develop techniques focused on specific tasks, resulting in a lack of general approaches that provide solutions that are transferable to different object types and tasks. Robots in assistive environments, able to manipulate cloth-like objects, would be a major step, able to provide new business opportunities.
In CHLOE-GRAPH we argue that the solution must embrace the different robotic layers, perception, decision making, and execution. We propose to develop a novel CHLOE (ClotH-Like ObjEcts) scene state representation (SSR) carefully designed to simplify its recognition, but including detailed information on the parts that have to be manipulated to execute the next action and also semantic tags designed for high-level planning.


The 4 objectives of CHLOE-GRAPH are

(O1) - To define the CHLOE-SSR by selecting semantic tags and parameters depending on the requirements from perception, execution, and high-level planning of manipulation tasks with textiles. The representation will have a central role: it will help to determine what has to be perceived, will be used to bootstrap planning operators, and will define the desired abilities of the manipulation actions.

(O2) To develop a CHLOE-SSR recognition method based on deep learning from real data labelled with the novel representation tags. We plan to combine computational topological methods with deep learning architectures to identify only the relevant features of the 2D surface in a 3D world, such as the environment contacts, the grasp location, the deformation category, and other manipulation affordance related information. We propose that only the relevant parts for manipulation need to be understood, not the full deformation state.

(O3) To implement a set of low-level primitives with strategies to accomplish transitions from state to state using the information in CHLOE-SSR. Depending on the complexity of each transition, we will use learning from demonstration techniques, end-to-end deep learning sim-to-real methods with dynamics extending from previous results, or model predictive control, implementing the parameters of the SSR within the motion characterization to allow for trajectory adaptability to the environment.

(O4) To develop a high-level decision-making framework to decide robot actions among the alternative plans, taking into account the non-complete observability of the state and the uncertainty on the action execution. We will contribute to three challenges: the representation, the complexity related to the high-low level gap, and the use of experiences to refine the learned transition model.
We will also propose novel measures to benchmark cloth manipulation tasks related to the segmentation steamed from the new representation, and demonstrate the validity of our approach in two robotic demonstrators.

Project Publications

Journal Publications

  • A. Luque, D. Parent, A. Colomé, C. Ocampo-Martínez and C. Torras. Model predictive control for dynamic cloth manipulation: Parameter learning and experimental validation. IEEE Transactions on Control Systems Technology: 1-17, 2024, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • D. Blanco-Mulero, O. Barbany, G. Alcan, A. Colomé, C. Torras and V. Kyrki. Benchmarking the sim-to-real gap in cloth manipulation. IEEE Robotics and Automation Letters, 9(3): 2981-2988, 2024.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Canal, C. Torras and G. Alenyà. Generating predicate suggestions based on the space of plans: an example of planning with preferences. User Modeling and User-Adapted Interaction, 33: 333-357, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • F. Coltraro, J. Fontana, J. Amorós, M. Alberich-Carramiñana, J. Borràs and C. Torras. A representation of cloth states based on a derivative of the Gauss Linking Integral. Applied Mathematics and Computation, 457: 128165, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. San-Miguel, V. Puig and G. Alenyà. Condition-based design of variable impedance controllers from user demonstrations. Control Engineering Practice, 140: 105658, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, S. Foix, S. Borgo and G. Alenyà. OCRA – An ontology for collaborative robotics and adaptation. Computers in Industry, 132: 103627, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs, B. Calli, A. Norton and G. Alenyà. Household cloth object set: Fostering benchmarking in deformable object manipulation. IEEE Robotics and Automation Letters, 7(3): 5866-5873, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • G. Tzelepis, A. Eren Erdal, J. Borràs and G. Alenyà. Semantic state estimation in robot cloth manipulations using domain adaptation from human demonstrations, 19 International Conference on Computer Vision Theory and Applications, 2024, Rome, pp. 172-182.

    Open/Close abstract Abstract Info Info pdf PDF

  • T. Love, A. Andriella and G. Alenyà. Towards explainable proactive robot interactions for groups of people in unstructured environments, 2024 ACM/IEEE International Conference on Human-Robot Interaction, 2024, Boulder, CO, USA, pp. 697–701.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Civit, A. Andriella, C. Barrue, M. Antonio, C. Boqué and G. Alenyà. Introducing social robots to assess frailty in older adults, 2024 ACM/IEEE International Conference on Human-Robot Interaction, 2024, Boulder, CO, USA, pp. 342–346.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Borràs, A. Boix, S. Foix and C. Torras. A virtual reality framework for fast dataset creation applied to cloth manipulation with automatic semantic labelling, 2023 IEEE International Conference on Robotics and Automation, 2023, London (UK), pp. 11605-11611.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Olivares-Alarcos, A. Andriella, S. Foix and G. Alenyà. Robot explanatory narratives of collaborative and adaptive experiences, 2023 IEEE International Conference on Robotics and Automation, 2023, London (UK), pp. 11964-11971.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Izquierdo, G. Alenyà and C. Rizzo. Adaptive Human-Robot Collaboration: evolutionary learning of action costs using an action outcome simulator, 32nd IEEE International Symposium on Robot and Human Interactive Communication, 2023, Busan, Korea, pp. 1901-1907.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Cupec, I. Vidovic, V. Simundic, P. Pejic, S. Foix and G. Alenyà. Teaching a robot where doors and drawers are and how to handle them, 32nd IEEE International Symposium on Robot and Human Interactive Communication, 2023, Busan, Korea, pp. 2288-2294.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Suárez, A. Andriella, C. Torras and G. Alenyà. User interactions and negative examples to improve the learning of semantic rules in a cognitive exercise scenario, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2023, Detroit, MI, USA, pp. 7953-7960.

    Open/Close abstract Abstract Info Info pdf PDF

  • G. Tzelepis, J. Borràs, A. Eren Erdal and G. Alenyà. Semantic state prediction in robotic cloth manipulation, 2023 Intelligent Systems Conference, 2023, Amsterdam, pp. 205-219, Springer Lecture Notes in Networks and Systems.

    Open/Close abstract Abstract Info Info pdf PDF

  • S. Izquierdo, G. Canal, C. Rizzo and G. Alenyà. Improved task planning through failure anticipation in human-robot collaboration, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 7875-7880.

    Open/Close abstract Abstract Info Info pdf PDF

  • I. Garcia-Camacho, J. Borràs and G. Alenyà. Knowledge representation to enable high-level planning in cloth manipulation tasks, 2022 ICAPS Workshop on Knowledge Engineering for Planning and Scheduling, 2022, Singapore (Virtual), pp. 9.

    Open/Close abstract Abstract Info Info pdf PDF

  • B.M. Lopes, R. Marques , T. Santos , J. Sousa and G. Alenyà. Relevant parameters identification in traditional & stretch and blow thermoplastics injection molding, 10th IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, 2022, Chemnitz, Germany , pp. 1-6, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • B.M. Lopes, J. Sousa and G. Alenyà. Machine learning methods for quality prediction in thermoplastics injection molding, 2021 IEEE International Conference on Electrical, Computer and Energy Technologies, 2021, Cape Town, South Africa (Virtual), pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF

Other Publications

  • A. Olivares-Alarcos, S. Foix and G. Alenyà. Time to contact for robot safety stop in close collaborative tasks. In Human-robot collaboration: unlocking the potential for industrial applications, 87-104. Institution of Engineering Technology, 2023.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Jiménez. Inside-out states of garments. Technical Report IRI-TR-24-01, Institut de Robòtica i Informàtica Industrial, CSIC-UPC, 2024.

    Open/Close abstract Abstract Info Info pdf PDF