Affiliations: Amirkabir University of Technology, Tehran, Iran
Correspondence:
[*]
Corresponding author: Mohammad Ebrahim Shiri, Amirkabir University of Technology, Tehran, Iran. Tel.: +98 21 6454 2548; E-mail: shiri@aut.ac.ir.
Abstract: State abstraction and value function approximation are powerful and useful methods for time and memory management in reinforcement learning. In traditional trends, these methods are applied to speed up learning of the current task; however, when we learn multiple similar environments in a general setting, these methods can be applied to improve learning of other tasks. We propose a framework to aggregate the results of state abstraction and function approximation in several tasks of a domain to reuse them in future tasks of that domain. First, we show theoretically how abstraction based on optimal value functions speeds up learning in that same task in the future. In many situations, fuzzy clustering is more natural than hard clustering, since it does not force the states to fully belong to one of the classes. In second part, we examine theoretically and algorithmically how using the knowledge extracted by fuzzy value approximation of a single task improves learning of that same task in the future. In both parts, we show that aggregating states (hard or fuzzy) preserves the optimal value function in the abstract space with an error bound. Having the support provided by these two parts, we propose new ways to combine the results of abstraction and approximation of different tasks of the domain to infer similarity measures on the state space. Finally, we show empirically that batch learning based on these similarity measures can speed up learning in the future tasks of the setting.
Keywords: Reinforcement learning, state abstraction, function approximation, transfer learning, fuzzy clustering
