Leaf segmentation and leaf modelling from color and ToF data for robotized plant measuring

Movie showing the successful probing of a plant leaf. Currently, we are developing a procedure for the automated probing of leaves.

In the context of the EU project GARNICS (GARNICS) we are interested in the segmentation of plants into their composite structures, i.e., leaves or part of leaves, and to extract color and 3D shape descriptors with which the robot can interact. In general, the identification and segmentation of 3D surfaces from an image is an important step towards solving object-manipulation tasks as it facilitates object recognition and grasp point selection, and, in consequence, the execution of appropriate grasping movements. We developed an algorithm that fuses color information and depth information from a time-of-flight depth camera (PMD CamCube) and uses this data to segment images into 3D surfaces. The robot arm used in the experiments is shown in Figure 1. Typical segmentation results of color-depth images of plants are shown Figure 2.

Figure 1: Robot arm at the IRI for plant measuring. A time-of-flight camera, a color camera, and a probing tool are mounted on the robot arm, exploring a plant.

Figure 2: Segmentation results (middle panel) and interpolated depth (right panel) for plant images.

We use this framework at the Institut de Robotica i Informatica Industrial to find good candidate leaves for probing. Our experimental set-up is shown in Figure 1 and a flow diagram of the method is shown in Figure 3.

Figure 3: Flow diagram of robotic plant measuring procedure.

The time-of-flight camera and color camera are mounted on a robotic arm. The stick is a placeholder for a measurement device that will be inserted later. A candidate leaf is selected and the 3D surface information associated with the leaf is used to compute a new robot position, which moves the camera system closer to the leaf (active vision). In robotic experiments, candidate leaves have been approached by the robot in order to validate their suitability to being sampled.

This work has been conducted at the Institut de Robotica i Informatica Industrial together with Prof. Carme Torras, Dr. Guillem Alenya, and Sergi Foix. More details can be found in Dellen et al (WACV 2011) and Alenya et al (ICRA 2011 and IEEE Robotics and Automation Magazine (2013)).