Subject:
|
Aligned constructions in machine translation Language & Speech Technology Language in Society Nederlab Nederlab |
Organization:
|
Taalwetenschap Communicatie- en informatiewetenschappen |
Journal title:
|
Journal of Open Research Software
|
Abstract:
|
Counting n-grams lies at the core of any frequentist corpus analysis and is often considered a trivial matter. Going beyond consecutive n-grams to patterns such as skipgrams and flexgrams increases the demand for efficient solutions. The need to operate on big corpus data does so even more. Lossless compression and non-trivial algorithms are needed to lower the memory demands, yet retain good speed. Colibri Core is software for the efficient computation and querying of n-grams, skipgrams and flexgrams from corpus data. The resulting pattern models can be analysed and compared in various ways. The software offers a programming library for C++ and Python, as well as command-line tools.
|