Cache-based Statistical Language Models of English and Highly Inflected Lithuanian

Article type: Research Article

Authors: Vaičiūnas, Airenas | Raškinis, Gailius

Affiliations: Department of Applied Informatics, Vytautas Magnus University, Vileikos 8, LT-44404 Kaunas, Lithuania, e-mail: airenas@freemail.lt, g.raskinis@if.vdu.lt

Abstract: This paper investigates a variety of statistical cache-based language models built upon three corpora: English, Lithuanian, and Lithuanian base forms. The impact of the cache size, type of the decay function, including custom corpus derived functions, and interpolation technique (static vs. dynamic) on the perplexity of a language model is studied. The best results are achieved by models consisting of 3 components: standard 3-gram, decaying cache 1-gram and decaying cache 2-gram that are joined together by means of linear interpolation using the technique of dynamic weight update. Such a model led up to 36% and 43% perplexity improvement with respect to the 3-gram baseline for Lithuanian words and Lithuanian word base forms respectively. The best language model of English led up to a 16% perplexity improvement. This suggests that cache-based modeling is of greater utility for the free word order highly inflected languages.

Keywords: language models, n-grams, cache models, dynamic interpolation, perplexity reduction, inflected language, free word order language, Lithuanian

Journal: Informatica, vol. 17, no. 1, pp. 111-124, 2006