Fine-Tuning LLMs for Croatian Text Retrieval

We investigate the possibility of adapting large language models (LLMs) to generate high-quality text embeddings in Croatian, a low-resource language. Text embeddings are vector representations of text used in retrieval tasks, where the goal is to find semantically similar documents based on vector distances rather than relying on exact keyword matching. Therefore, generating quality text embeddings is a key component of non-keyword-based text retrieval systems.
In our study, we fine-tune several LLMs such as Mistral, LLaMA, and Gemma, as well as embedding models, including BGE, using parameter-efficient fine-tuning. We identify BGE-HR, our adapted version of BGE, as the best-performing model. Rather than focusing solely on introducing a new model, we conduct a systematic analysis of the factors that influence its successful adaptation to low-resource languages. An important aspect of this analysis involves evaluating the ability of base LLMs to generate coherent text in Croatian, including how effectively they segment text into smaller units (tokens) for subsequent processing.
To further assess model generalizability, we introduce a new evaluation dataset derived from commonly asked questions found on the Croatian web. This enables us to benchmark performance on realistic text retrieval tasks that differ from the dataset used for fine-tuning. Our findings suggest that choosing a stronger base model is generally more effective than investing substantial effort in fine-tuning a weaker one. We also find that some smaller, publicly available multilingual embedding models, generalize well on the new dataset, offering a practical solution when computational resources for fine-tuning are limited.