Tehničko veleučilište u Zagrebu · Zagreb

An economic approach to big data in a minority language

izvorni znanstveni rad

izvorni znanstveni rad

An economic approach to big data in a minority language

Vrsta prilog u časopisu
Tip izvorni znanstveni rad
Godina 2014
Časopis Procedia computer science
Nadređena publikacija Procedia Computer Science
Volumen 35
Stranice str. 427-436
DOI 10.1016/j.procs.2014.08.123
EISSN 1877-0509
Status objavljeno

Sažetak

Google's n-gram project brought recently big data benefits to several main world languages, like English, Chinese etc. Any attempt to derive such systems, aimed to accelerate the development of NLP applications for world minority languages, in the manner in which it has been done in the project, encounters many obstacles. This paper presents an innovative and economic approach to large-scale n-gram system creation applied to the Croatian language case. Instead of using the Web as the world's biggest text repository, our process of n-gram collection relies on the Croatian academic online spellchecker Hascheck, a language service publicly available since 1993 and popular worldwide. The service has already processed a corpus whose size exceeds the size of the Croatian web-corpus created in recent years. Contrary to the Google n-gram systems, where cutoff criteria were applied, our n-gram filtering is based on dictionary criteria. This resulted in a system comparable in size to the largest n-gram systems of today. Because of the reliance on a service in constant use, the Croatian n-gram system is a dynamic one, unique among the systems compared. The importance of having an n-gram infrastructure for rapid breakthroughs in new application areas is also exemplified in the paper.

Ključne riječi

Croatian language; language modeling; lexical n - gram; n - gram count comparison; traffic modeling .