An Influence of Nonlinearities to Storage Capacity of Neural Networks

Article type: Research Article

Authors: Garliauskas, Algis

Affiliations: Institute of Mathematics and Informatics, Akademijos 4, LT‐08663, Vilnius, Lithuania, e‐mail: galgis_1@ktl.mii.lt

Abstract: The more realistic neural soma and synaptic nonlinear relations and an alternative mean field theory (MFT) approach relevant for strongly interconnected systems as a cortical matter are considered. The general procedure of averaging the quenched random states in the fully‐connected networks for MFT, as usually, is based on the Boltzmann Machine learning. But this approach requires an unrealistically large number of samples to provide a reliable performance. We suppose an alternative MFT with deterministic features instead of stochastic nature of searching a solution a set of large number equations. Of course, this alternative theory will not be strictly valid for infinite number of elements. Another property of generalization is an inclusion of the additional member in the effective Hamiltonian allowing to improve the stochastic hill‐climbing search of the solution not dropping into local minima of the energy function. Especially, we pay attention to increasing of neural networks retrieval capability transforming the replica‐symmetry model by including of different nonlinear elements. Some results of numerical modeling as well as the wide discussion of neural systems storage capacity are presented.

Keywords: mean field theory, storage capacity, nonlinearity, neural networks

Journal: Informatica, vol. 16, no. 2, pp. 159-174, 2005