IntellAct: Intelligent observation and execution of Actions and manipulations

Project description

IntellAct addresses the problem of understanding and exploiting the meaning (semantics) of manipulations in terms of objects, actions and their consequences for reproducing human actions with machines. It comprises three building blocks: (1) Learning: Abstract, semantic descriptions of manipulations are extracted from video sequences showing a human demonstrating the manipulations; (2) Monitoring: In the second step, observed manipulations are evaluated against the learned, semantic models; (3) Execution: Based on learned, semantic models, equivalent manipulations are executed by a robot. The analysis of low-level observation data for semantic content (Learning) and the synthesis of concrete behavior (Execution) constitute the major scientific challenge of IntellAct. Based on the semantic interpretation and description and enhanced with low-level trajectory data for grounding, two major application areas are addressed by IntellAct: First, the monitoring of human manipulations for correctness (e.g., for training or in high-risk scenarios) and second, the efficient teaching of cognitive robots for manipulation (e.g. in the context of assembly processes in industrial settings and on a space station). To achieve these goals, IntellAct brings together recent methods for (1) parsing scenes into spatio-temporal graphs and so called ‘semantic event chains’, (2) probabilistic models of objects and their manipulation, (3) probabilistic rule learning, and (4) dynamic motion primitives for trainable and flexible descriptions of robotic motor behavior. Its implementation employs a concurrent-engineering approach that includes virtual-reality-enhanced simulation as well as physical robots. Its goal culminates in the demonstration of a robot understanding, monitoring and reproducing human action.

Publications