arxiv6d ago
MixLID: Token-Level Language Identification in Mixed-Language Text
arXiv:2610.16601v1 Announce Type: cross Abstract: Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 l