Research Project

CHLOE-GRAPH: ClotH-Like ObjEcts Grasping, Representation, and Action Planning


National Project

Start Date


End Date


Project Code


Project illustration


Project Description

Project PID2020-118649RB-I00 funded by MCIN/ AEI /10.13039/501100011033

CHLOE-GRAPH proposes a novel robotic manipulation approach for cloth manipulation that is expected to be general enough to be applied to different assistive tasks. While rigid object manipulation has achieved a decent maturity, robots are yet to make an impact in the handling of deformable objects. Cloth-like objects are extremely challenging due to the complexity of understanding their deformation often with severe self-occlusions, making learning from real data very difficult. This has led to a tendency to develop techniques focused on specific tasks, resulting in a lack of general approaches that provide solutions that are transferable to different object types and tasks. Robots in assistive environments, able to manipulate cloth-like objects, would be a major step, able to provide new business opportunities.
In CHLOE-GRAPH we argue that the solution must embrace the different robotic layers, perception, decision making, and execution. We propose to develop a novel CHLOE (ClotH-Like ObjEcts) scene state representation (SSR) carefully designed to simplify its recognition, but including detailed information on the parts that have to be manipulated to execute the next action and also semantic tags designed for high-level planning.

The 4 objectives of CHLOE-GRAPH are

(O1) - To define the CHLOE-SSR by selecting semantic tags and parameters depending on the requirements from perception, execution, and high-level planning of manipulation tasks with textiles. The representation will have a central role: it will help to determine what has to be perceived, will be used to bootstrap planning operators, and will define the desired abilities of the manipulation actions.

(O2) To develop a CHLOE-SSR recognition method based on deep learning from real data labelled with the novel representation tags. We plan to combine computational topological methods with deep learning architectures to identify only the relevant features of the 2D surface in a 3D world, such as the environment contacts, the grasp location, the deformation category, and other manipulation affordance related information. We propose that only the relevant parts for manipulation need to be understood, not the full deformation state.

(O3) To implement a set of low-level primitives with strategies to accomplish transitions from state to state using the information in CHLOE-SSR. Depending on the complexity of each transition, we will use learning from demonstration techniques, end-to-end deep learning sim-to-real methods with dynamics extending from previous results, or model predictive control, implementing the parameters of the SSR within the motion characterization to allow for trajectory adaptability to the environment.

(O4) To develop a high-level decision-making framework to decide robot actions among the alternative plans, taking into account the non-complete observability of the state and the uncertainty on the action execution. We will contribute to three challenges: the representation, the complexity related to the high-low level gap, and the use of experiences to refine the learned transition model.
We will also propose novel measures to benchmark cloth manipulation tasks related to the segmentation steamed from the new representation, and demonstrate the validity of our approach in two robotic demonstrators.

Project Publications

Journal Publications

  • A. Olivares-Alarcos, S. Foix, S. Borgo and G. Alenyà. OCRA – An ontology for collaborative robotics and adaptation. Computers in Industry, 132: 103627, 2022.

    Open/Close abstract Abstract Info Info pdf PDF
  • I. Garcia-Camacho, J. Borràs, B. Calli, A. Norton and G. Alenyà. Household cloth object set: Fostering benchmarking in deformable object manipulation. IEEE Robotics and Automation Letters, 7(3): 5866-5873, 2022, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Canal, C. Torras and G. Alenyà. Generating predicate suggestions based on the space of plans: an example of planning with preferences. User Modeling and User-Adapted Interaction, 2022, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • S. Izquierdo, G. Canal, C. Rizzo and G. Alenyà. Improved task planning through failure anticipation in human-robot collaboration, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 7875-7880.

    Open/Close abstract Abstract Info Info pdf PDF
  • I. Garcia-Camacho, J. Borràs and G. Alenyà. Knowledge representation to enable high-level planning in cloth manipulation tasks, 2022 ICAPS Workshop on Knowledge Engineering for Planning and Scheduling, 2022, Singapore (Virtual), pp. 9.

    Open/Close abstract Abstract Info Info pdf PDF
  • B.M. Lopes, J. Sousa and G. Alenyà. Machine learning methods for quality prediction in thermoplastics injection molding, 2021 IEEE International Conference on Electrical, Computer and Energy Technologies, 2021, Cape Town, South Africa (Virtual), pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF