On-line Approximate String Matching in Natural Language

Article type: Research Article

Authors: Fredriksson, Kimmo

Affiliations: Department of Computer Science, University of Joensuu, PO Box 111, 80101 Joensuu, Finland. E-mail: kfredrik@cs.joensuu.fi

Abstract: We consider approximate pattern matching in natural language text. We use the words of the text as the alphabet, instead of the characters as in traditional string matching approaches. Hence our pattern consists of a sequence of words. From the algorithmic point of view this has several advantages: (i) the number of words is much less than the number of characters, which in effect means shorter text (less possible matching positions); (ii) the pattern effectively becomes shorter, so bit-parallel techniques become more applicable; (iii) the alphabet size becomes much larger, so the probability that two symbols (in this case, words) match is reduced. We extend several known approximate string matching algorithms for this scenario, allowing k insertions, deletions or substitutions of symbols (natural language words). We further extend the algorithms to allow k' errors inside the pattern symbols (words) as well. The two error thresholds k and k' can be applied simultaneously and independently. Hence we have in effect two alphabets, and perform approximate matching in both levels. From the application point of view the advantage is that the method is flexible, allowing simple solutions to problems that are hard to solve with traditional approaches. Finally, we extend the algorithms to handle multiple patterns at the same time. Depending on the search parameters, we obtain algorithms that run in linear or sublinear time and that perform the optimal number of word comparisons on average, We conclude with experimental results showing that the methods work well in practice.

Keywords: Approximate string matching, filtering, natural language processing, phrase matching, word alphabet

Journal: Fundamenta Informaticae, vol. 72, no. 4, pp. 453-466, 2006