Affiliations: [a] Transport Systems Planning and Transport Telematics, Technische Universität Berlin, Germany. E-mail: tziemke@vsp.tu-berlin.de | [b] Instituto da Informática, Universidade Federal do Rio Grande do Sul (UFRGS), Brazil. E-mails: lnalegre@inf.ufrgs.br, bazzan@inf.ufrgs.br
Abstract: Reinforcement learning is an efficient, widely used machine learning technique that performs well when the state and action spaces have a reasonable size. This is rarely the case regarding control-related problems, as for instance controlling traffic signals. Here, the state space can be very large. In order to deal with the curse of dimensionality, a rough discretization of such space can be employed. However, this is effective just up to a certain point. A way to mitigate this is to use techniques that generalize the state space such as function approximation. In this paper, a linear function approximation is used. Specifically, SARSA(λ) with Fourier basis features is implemented to control traffic signals in the agent-based transport simulation MATSim. The results are compared not only to trivial controllers such as fixed-time, but also to state-of-the-art rule-based adaptive methods. It is concluded that SARSA(λ) with Fourier basis features is able to outperform such methods, especially in scenarios with varying traffic demands or unexpected events.
Keywords: Reinforcement learning, traffic signal control, linear function approximation, transport simulation
