Quadratic optimization for the hyper-parameter based on maximum entropy search

Article type: Research Article

Authors: Li, Yuqi^{; *}

Affiliations: School of Economics and Management, Beijing University of Posts and Telecommunications, Beijing, China

Correspondence: [*] Corresponding author. Yuqi Li. E-mail: liyuqi0033@sina.com.

Abstract: The method based on entropy was used for the Bayesian optimization. Based on compelling information theory, Entropy Search (ES) and Predictive Entropy Search (PES) maximized information about the unknown function when the loss function reaches the maximum value. However, both methods were plagued by complicated calculations for estimating entropy. The most important motivation of this article is to improve and modularize the entropy search itself, making this method more flexible and effective for model adaptation. After the initial optimization and pruning module processing, a reasonable initial configuration for the complex model was successfully established, further reducing the space required for secondary optimization hyper-parameter search. The advantage of this method is that, on the one hand, the basic method of Bayesian optimization is used to get the best result of the iteration, while ensuring that the algorithm has theoretical boundedness. On the other hand, through the maximum entropy, the information features of the original space and data set are retained as much as possible to reduce the loss of information due to the initialization process, so as to improve the precision of the secondary optimization of the model. Further, a new algorithm framework is proposed, integrating MES and Sequential Model-Based Optimization (SMBO). With MES as the final module of the whole optimization process, a more accurate and reasonable algorithmic model was built, which lays a solid mathematical basis for the final empirical analysis.

Keywords: Bayesian optimization, SMBO, hyperparameter optimization, entropy search

DOI: 10.3233/JIFS-230470

Journal: Journal of Intelligent & Fuzzy Systems, vol. 45, no. 3, pp. 4991-5006, 2023