09/26 Alec Koppel (U.S. Army Research Laboratory), Policy Search for Reinforcement Learning in Continuous Spaces: Improved Limits and Reduced Variance

ABSTRACT: Reinforcement Learning (RL) is a form of stochastic adaptive control in which one seeks to estimate parameters of a controller without having access to a dynamics model. RL has gained popularity in recent years beginning with the smashing success of AlphaGo besting the world champion in Go during summer 2016. However, the recent empirical successes of RL have been called into question due to their irreproducibility and high variance across different training runs. Motivated by this gap, we'll spotlight recent efforts to solidify theoretical understanding of the rate analysis and limiting properties of policy gradient methods in continuous Markov Decision Problems from a non-convex optimization perspective. Moreover, we design modified step-size rules that yield convergence to approximate local extrema, motivating reward-reshaping via nonconvex optimization. We'll then discuss a modification of the Policy Gradient Theorem that yields provably lower variance policy search directions, and algorithms based upon which yield algorithms with reduced variance. These results provide a conceptual framework for the future design of stable RL tools with lower variance.