On Stochastic Optimization and Statistical Learning in Reproducing Kernel Hilbert Spaces by Support Vector Machines (SVM)

Article type: Research Article

Authors: Norkin, Vladimir | Keyzer, Michiel

Affiliations: Glushkov Institute of Cybernetics, National Academy of Sciences of Ukraine, Glushkov avenue 40, 03187 Kiev, Ukraine, e-mail: norkin@i.com.ua | Center for World Food Studies (SOW-VU), VU University of Amsterdam, De Boelelaan 1105, 1081 HV, Amsterdam, The Netherlands, e-mail: m.a.keyzer@sow.vu.nl

Abstract: The paper studies stochastic optimization problems in Reproducing Kernel Hilbert Spaces (RKHS). The objective function of such problems is a mathematical expectation functional depending on decision rules (or strategies), i.e. on functions of observed random parameters. Feasible rules are restricted to belong to a RKHS. This kind of problems arises in on-line decision making and in statistical learning theory. We solve the problem by sample average approximation combined with Tihonov's regularization and establish sufficient conditions for uniform convergence of approximate solutions with probability one, jointly with a rule for downward adjustment of the regularization factor with increasing sample size.

Keywords: stochastic optimization, empirical risk minimization, decision rule, Reproducing Kernel Hilbert Spaces (RKHS), support vector machine (SVM), regularization, convergence

Journal: Informatica, vol. 20, no. 2, pp. 273-292, 2009