Publication
Dual REPS: A generalization of relative entropy policy search exploiting bad experiences
Journal Article (2017)
Journal
IEEE Transactions on Robotics
Pages
978-985
Volume
33
Number
4
Doc link
http://dx.doi.org/10.1109/TRO.2017.2679202
File
Abstract
Policy Search (PS) algorithms are nowadays widely used for their simplicity and effectiveness in finding solutions for robotic problems. However, most current PS algorithms derive policies by statistically fitting the data from the best experiments only. This means that those experiments yielding a poor performance are usually discarded or given too little influence on the policy update. In this paper, we propose a generalization of the Relative Entropy Policy Search (REPS) algorithm that takes bad experiences into consideration when computing a policy. The proposed approach, named Dual REPS (DREPS) following the philosophical interpretation of the duality between good and bad, finds clusters of experimental data yielding a poor behavior and adds them to the optimization problem as a repulsive constraint. Thus, considering there is a duality between good and bad data samples, both are taken into account in the stochastic search for a policy. Additionally, a cluster with the best samples may be included as an attractor to enforce faster convergence to a single optimal solution in multi-modal problems. We first tested our proposed approach in a simulated Reinforcement Learning (RL) setting and found that DREPS considerably speeds up the learning process, especially during the early optimization steps and in cases where other approaches get trapped in between several alternative maxima. Further experiments in which a real robot had to learn a task with a multi-modal reward function confirm the advantages of our proposed approach with respect to REPS.
Categories
learning (artificial intelligence), manipulators, stochastic programming.
Author keywords
Reinforcement Learning, Direct Policy Search, Learning from Demonstration
Scientific reference
A. Colomé and C. Torras. Dual REPS: A generalization of relative entropy policy search exploiting bad experiences. IEEE Transactions on Robotics, 33(4): 978-985, 2017.
Follow us!