Sentence Level Alignment of Digitized Books Parallel Corpora

Article type: Research Article

Authors: Laukaitis, Algirdas^{1; *} | Plikynas, Darius² | Ostasius, Egidijus¹

Affiliations: [1] Vilnius Gediminas Technical University, Fundamental Science Faculty | [2] Vilnius University, Institute of Data Science and Digital Technologies. E-mails: algirdas.laukaitis@vgtu.lt, darius.plikynas@mii.vu.lt, egidijus.ostasius@vgtu.lt

Correspondence: [*] Corresponding author.

Abstract: In this paper, we propose a framework for extracting translation memory from a corpus of fiction and non-fiction books. In recent years, there have been several proposals to align bilingual corpus and extract translation memory from legal and technical documents. Yet, when it comes to an alignment of the corpus of translated fiction and non-fiction books, the existing alignment algorithms give low precision results. In order to solve this low precision problem, we propose a new method that incorporates existing alignment algorithms with proactive learning approach. We define several feature functions that are used to build two classifiers for text filtering and alignment. We report results on English-Lithuanian language pair and on bilingual corpus from 200 books. We demonstrate a significant improvement in alignment accuracy over currently available alignment systems.

Keywords: alignment of corpora, alignment of digitized books, machine translation, natural language processing

Journal: Informatica, vol. 29, no. 4, pp. 693-710, 2018