HF RL Explorer

Loading a SentencePiece tokenizer with byte fallback as a fast tokenizer hard-fails

Loading a SentencePiece tokenizer with byte fallback as a fast tokenizer hard-fails: a task in MiMo-V2.6-RL-oss: Agentic RL Environments (MiMo RL release). Loading a SentencePiece tokenizer with byte fallback as a fast tokenizer hard-fails After updating to a recent transformers version…

The task

Loading a SentencePiece tokenizer with byte_fallback as a fast tokenizer hard-fails After updating to a recent transformers version, converting a slow SentencePiece tokenizer that was trained with the byte_fallback option into its fast counterpart no longer works at all — it…

Part of XiaomiMiMo/MiMo-V2.6-RL-oss.