Note: [1] This work was supported in part by the National Natural Science Foundation of China (Grant No. 61471252).
Abstract: Reinforcement learning is hard to solve optimization problems in multi-agent system because of the inefficiency of function approximation. Sparse distributed memories, which is implemented using Radial Basis Functions or Kanerva Coding, can be used to improve the efficiency. But this approach still often gives poor performance when applied to large-scale multi-agent systems. In this paper, we attempt to solve four-rooms problem in the predator-prey pursuit domain and argue that the poor performance that we observe is caused by frequent prototype collisions. We show that dynamic prototype allocation and adaptation can give better results by reducing these collisions. By using our novel approach about fuzzy Kanerva-based function approximation, that uses a fine-grained fuzzy membership grade to describe a state-action pair’s adjacency with respect to each prototype, we give some results that prototype collisions are completely eliminated and learning performance is greatly improved. We further show that prototype density varies widely across the state-action space and that this variation causes prototypes’ receptive fields to be unevenly distributed. This distribution limits the ability of fuzzy Kanerva Coding to achieve better results. It can be observed that fuzzy Kanerva Coding allows prototypes to adaptively tune their receptive fields for a target application. We conclude that fuzzy Kanerva Coding with prototype tuning and adaptation can significantly improve a reinforcement learner’s ability to solve large-scale multi-agent problems.
Keywords: Reinforcement learning, function approximation, fuzzy system, pursuit
problem
