A Virtual Reality Framework For Fast Dataset Creation Applied to Cloth Manipulation with Automatic Semantic Labelling

Júlia Borràs, Arnau Boix-Granell, Sergi Foix, and Carme Torras

Institut de Robòtica i Informàtica Industrial, CSIC-UPC, C/ Llorens i Artigas 4-6, 08028 Barcelona, Spain.

Abstract - Teaching complex manipulation skills, such as folding garments, to a bi-manual robot is a very challenging task, which is often tackled through learning from demonstration. The few datasets of garment-folding demonstrations available nowadays to the robotics research community have been either gathered from human demonstrations or generated through simulation. The former have the great difficulty of perceiving both cloth state and human action as well as transferring them to the dynamic control of the robot, while the latter require coding human motion into the simulator in open loop, i.e., without incorporating the visual feedback naturally used by people, resulting in far-from-realistic movements. In this article, we present an accurate dataset of human cloth folding demonstrations. The dataset is collected through our novel virtual reality (VR) framework, based on Unity’s 3D platform and the use of an HTC Vive Pro system. The framework is capable of simulating realistic garments while allowing users to interact with them in real time through handheld controllers. By doing so, and thanks to the immersive experience, our framework permits exploiting human visual feedback in the demonstrations while at the same time getting rid of the difficulties of capturing the state of cloth, thus simplifying data acquisition and resulting in more realistic demonstrations. We create and make public a dataset of cloth manipulation sequences, whose cloth states are semantically labeled in an automatic way by using a novel low-dimensional cloth representation that yields a very good separation between different cloth configurations.


Examples of predicted cloth states

Dataset manipulations graph

Fig. 1: Graph of state sequences, following the manipulation representation framework in [1]. The coloured dots indicate the different types of grasp types and grasp locations that can be performed to pass from the previous state to the next.



The research leading to these results receives funding from the European Research Council (ERC) from the European Union Horizon 2020 Programme under grant agreement no. 741930 (CLOTHILDE: CLOTH manIpulation Learning from DEmonstrations) and project SoftEnable (HORIZON-CL4-2021-DIGITAL-EMERGING-01-101070600). Authors also received funding from project CHLOE-GRAPH (PID2020-118649RB-I00) funded by MCIN/ AEI /10.13039/501100011033 and COHERENT (PCI2020-120718-2) funded by MCIN/ AEI /10.13039/501100011033 and cofunded by the ”European Union NextGenerationEU/PRTR”.


[1] I. Garcia-Camacho, J. Borràs, and G. Alenyà. Knowledge representation to enable high-level planning in cloth manipulation tasks. ICAPS 2022 Workshop on Knowledge Engineering for Planning and Scheduling, 2022.



author={Borràs, Júlia and Boix-Granell, Arnau and Foix, Sergi and Torras, Carme},
booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, London, UK},
title={A virtual reality framework for fast dataset creation applied to cloth manipulation with automatic semantic labelling},