Why neural networks in the first place: a theoretical explanation

Issue title: Soft Computing and Advances in Intelligent Systems

Guest editors: Ildar Batyrshin, Fernando Gomide, Vladik Kreinovich and Shahnaz Shahbazova

Article type: Research Article

Authors: Contreras, Jonatan^a | Ceberio, Martine^a | Kosheleva, Olga^b | Kreinovich, Vladik^{a; *}

Affiliations: [a] Department of Computer Science, University of Texas at El Paso, TX, USA | [b] Department of Teacher Education, University of Texas at El Paso, TX, USA

Correspondence: [*] Corresponding author. Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, 500 W. University, El Paso, TX 79968, USA. E-mail: vladik@utp.edu.

Abstract: Neural networks – specifically, deep neural networks – are, at present, the most effective machine learning techniques. There are reasonable explanations of why deep neural networks work better than traditional “shallow” ones, but the question remains: why neural networks in the first place? why not networks consisting of non-linear functions from some other family of functions? In this paper, we provide a possible theoretical answer to this question: namely, we show that of all families with the smallest possible number of parameters, families corresponding to neurons are indeed optimal – for all optimality criteria that satisfy some reasonable requirements: namely, for all optimality criteria which are final and invariant with respect to coordinate changes, changes of measuring units, and similar linear transformations.

Keywords: Neural networks, invariance, function approximation, theoretical explanation

DOI: 10.3233/JIFS-212009

Journal: Journal of Intelligent & Fuzzy Systems, vol. 43, no. 6, pp. 6947-6951, 2022