Automatic skill acquisition in reinforcement learning using graph centrality measures

Article type: Research Article

Authors: Moradi, Parham^a | Shiri, Mohammad Ebrahim^{a; *} | Rad, Ali Ajdari^b | Khadivi, Alireza^b | Hasler, Martin^b

Affiliations: [a] Department of Computer Science, Faculty of Mathematics & Computer Science, Amirkabir University of Technology, Tehran, Iran | [b] LANOS, IC, Ecole Polytechnique Federal de Lausanne, Lausanne, Switzerland

Correspondence: [*] Corresponding author: Mohammad Ebrahim Shiri, Department of Computer Science, Faculty of Mathematics & Computer Science, Amirkabir University of Technology, Tehran, Iran. Tel.: +98 21 64542548; Fax: +98 21 66497930; E-mail: shiri@aut.ac.ir

Abstract: Mechanisms on automatic discovery of macro actions or skills in reinforcement learning methods are mainly focused on subgoal discovery methods. Among the proposed algorithms, those based on graph centrality measures demonstrate a high performance gain. In this paper, we propose a new graph theoretic approach for automatically identifying and evaluating subgoals. Moreover, we propose a method for providing some useful prior knowledge for corresponding policy of developed skills based on two graph centrality measures, namely node connection graph stability and co-betweenness centrality. Investigating some benchmark problems, we show that the proposed approach improves the learning performance of the agent significantly.

Keywords: Hierarchical reinforcement learning, skill acquisition, graph centrality measures, node connection graph stability, prior knowledge injection

DOI: 10.3233/IDA-2011-0513

Journal: Intelligent Data Analysis, vol. 16, no. 1, pp. 113-135, 2012