Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core

Fulltext present in this item

Format: PDF

Description: publisher's version

Title:	Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core
Author(s):	Gompel, M. van ; Bosch, A.P.J. van den
Publication year:	2016
Source:	Journal of Open Research Software, vol. 4, iss. 1, (2016), pp. 1-10, article e30
Number of Pages:	10 p.
ISSN:	2049-9647
DOI:	https://doi.org/10.5334/jors.105
Publication type:	Article / Letter to editor
Please use this identifier to cite or link to this item : https://hdl.handle.net/2066/159818
Display more details
Subject:	Aligned constructions in machine translation Language & Speech Technology Language in Society Nederlab Nederlab
Organization:	Taalwetenschap Communicatie- en informatiewetenschappen
Journal title:	Journal of Open Research Software
Volume:	vol. 4
Issue:	iss. 1
Page start:	p. 1
Page end:	p. 10
Article number:	e30
Abstract:	Counting n-grams lies at the core of any frequentist corpus analysis and is often considered a trivial matter. Going beyond consecutive n-grams to patterns such as skipgrams and flexgrams increases the demand for efficient solutions. The need to operate on big corpus data does so even more. Lossless compression and non-trivial algorithms are needed to lower the memory demands, yet retain good speed. Colibri Core is software for the efficient computation and querying of n-grams, skipgrams and flexgrams from corpus data. The resulting pattern models can be analysed and compared in various ways. The software offers a programming library for C++ and Python, as well as command-line tools.
Languages used:	English (eng)