Affiliations: Department of Computer Science, University of Paderborn, Paderborn, Germany
Correspondence:
[*]
Corresponding author: Michael Baumann, Department of Computer Science, University of Paderborn, 33095 Paderborn, Germany. E-mail: mbaumann@upb.de
Note: [1] A shorter version [2] of this work appeared as “Adaptive Function Approximation in Reinforcement Learning with an Interpolating Growing Neural Gas” in Proceedings of the 12th International Conference on Hybrid Intelligent Systems (HIS), 2012, pp. 512–517.
Abstract: Q-Learning is a widely used method for dealing with reinforcement learning problems. To speed up learning and to exploit gained experience more efficiently it is highly beneficial to add generalization to Q-Learning and thus enabling the transfer of experience to unseen but similar states. In this paper, we report on improvements for GNG-Q, a combination of Q-Learning and growing neural gas (GNG). It solves reinforcement learning problems with continuous state spaces and simultaneously learns a proper approximation of the state space by starting with a coarse resolution that is gradually refined based on information achieved during learning. We introduce the Interpolating GNG-Q (IGNG-Q) that uses distance-based interpolation between learned Q-vectors, adjust the update rule, suggest a new refinement strategy and propose a new criterion to decide when a refinement is necessary. Furthermore, we argue that this criterion offers an implicit local stopping condition for changes made to the approximation. Additionally, we employ eligibility traces to speed up learning. The improved method is evaluated in continuous state spaces and the results are compared with several approaches from literature. Our experiments confirm that the modifications highly improve the efficiency of the approximation and that IGNG-Q is well competitive with existing methods.
Keywords: Growing neural gas, function approximation, continuous reinforcement learning
