Reverse token filter mangles strings that contain combining characters
Reverse token filter mangles strings that contain combining characters: a task in MiMo-V2.6-RL-oss: Agentic RL Environments (MiMo RL release). I'm using the reverse token filter as part of an analyzer chain to index text in a few languages that make heavy use of diacritics (think Vietnamese, or…
The task
I'm using the reverse token filter as part of an analyzer chain to index text in a few languages that make heavy use of diacritics (think Vietnamese, or French/Spanish text that ends up in NFD form after some upstream normalization). The idea is to reverse the token so I can do…
Part of XiaomiMiMo/MiMo-V2.6-RL-oss.