Enhancing Nash Q-learning and Team Q-learning mechanisms by using bottlenecks

Article type: Research Article

Authors: Ghazanfari, Behzad | Mozayani, Nasser

Affiliations: School of Computer Engineering, Iran university of Science and Technology, Tehran, Iran

Note: [] Corresponding author. Behzad Ghazanfari, School of Computer Engineering, Iran university of Science and Technology, Tehran, Iran. Tel.: +98 9379335907; Fax: +98 21 73225322; E-mail: beghazanfari@gmail.com

Abstract: Nash Q-learning and team Q-learning are extended versions of reinforcement learning method for using in Multi-agent systems as cooperation mechanisms. The complexity of multi-agent reinforcement learning systems is extremely high thus it is necessary to use complexity reduction methods like hierarchical structures, abstraction and task decomposition. A typical approach for the latter to define subtasks is based on extracting bottlenecks. In this paper, bottlenecks are automatically extracted to create temporally extended actions which are in turn added to available agent's actions in cooperation mechanisms of multi-agent systems. The updating equations of team Q-learning and Nash Q-learning are extended in such a way to involve temporally extended actions. In this way the performance of learning in team Q-learning and Nash Q-learning is considerably increased. The experimental results show an interesting improvement in the process of learning of cooperation mechanisms being augmented by extracted temporally actions in multi-agent problems.

Keywords: Reinforcement learning, bottlenecks, clustering, options, hierarchical reinforcement learning

DOI: 10.3233/IFS-130945

Journal: Journal of Intelligent & Fuzzy Systems, vol. 26, no. 6, pp. 2771-2783, 2014