Tokenizer breaks on SQL strings ending at exact buffer boundary

Tokenizer breaks on SQL strings ending at exact buffer boundary: a task in harbor-swesmith-rl-artifacts: Harbor SWE-Smith 强化学习数据产物 (Harbor dataset). The tokenizer fails to properly handle SQL strings that end exactly at the buffer boundary, causing incorrect tokenization behavior. This affects…

The task

The tokenizer fails to properly handle SQL strings that end exactly at the buffer boundary, causing incorrect tokenization behavior. This affects various SQL parsing scenarios where the input string length aligns with internal buffer management.

Part of keryszhan/harbor-swesmith-rl-artifacts.