Evaluating The Syntactic Knowledge of Language Models on Lithuanian
Urtė Jakubauskaitė

Abstract:
Despite substantial recent advances in language models (LMs), their syntactic competence remains largely unexplored in low-resource languages. Lithuanian presents a particularly interesting test case due to its rich morphology and the scarcity of evaluation resources. The only existing benchmark covers a limited range of syntactic phenomena, making it difficult to assess how well language models capture the complexities of Lithuanian grammar.
To address this gap, this thesis introduces a new minimal-pair benchmark for Lithuanian syntax. The dataset consists of grammatical and ungrammatical sentence pairs targeting 31 linguistic phenomena and 64 error types derived from attested language use. The benchmark is used to evaluate 78 monolingual and multilingual language models. In addition, human acceptability judgments are collected to enable comparisons between model predictions and native speaker intuitions.
The results reveal large variation in performance across models and syntactic constructions. Mono- lingual Lithuanian models achieve the strongest results, and performance generally improves with model size. Moreover, the proposed benchmark proves considerably more challenging than the existing multi- lingual benchmark including Lithuanian, exposing weaknesses that remained undetected in previous research. Overall, this work extends the available evaluation resources for Lithuanian and contributes to a more comprehensive assessment of syntactic competence in low-resource, morphologically rich languages.