Irene Garcia-Camacho, Júlia Borràs and Guillem Alenyà
Abstract - Benchmarking robot manipulations is complex due to the difficulty in reproducing and comparing results across different embodiments and scenarios. Cloth manipulation presents additional challenges due to the complex object configuration space, making it difficult to define initial poses, to find standard objects sets or assessing the quality of a task. Traditional cloth manipulation papers do not have well defined metrics to evaluate the success of a task or the quality of the result, and are tailored to each evaluation. In this paper we propose to evaluate cloth manipulation using action graphs that model a task as a graph of transitions between different scene states. This results into an action segmentation that allows us to evaluate each segment, its influence in the next segment and relate it to task as a whole. Using a common bi-manual cloth manipulation task such as placing a cloth flat on a table, we propose a benchmark with simple evaluation metrics that explore the influence of grasp location and trajectory into the quality of the task. Our results show that grasp location doesn’t need to be precise on corners, that quality measures focused on evaluating different cloth parts can enlighten issues to solve and that success definition of a segment has to consider its influence on the ability to perform successfully the next segment of action.
The repository to evaluate the placed cloth on top of the table is avaliable on Github with instructions on how to use it and examples of real results of the implemented baseline to test the packages.