HF RL Explorer

Duplicate URLs being fetched when redirects are involved ( AllowURLRevisit=false )

Duplicate URLs being fetched when redirects are involved ( AllowURLRevisit=false ): a task in MiMo-V2.6-RL-oss: Agentic RL Environments (MiMo RL release). Duplicate URLs being fetched when redirects are involved (AllowURLRevisit=false) I'm using colly to crawl a site and I'm relying on the default…

The task

Duplicate URLs being fetched when redirects are involved (AllowURLRevisit=false) I'm using colly to crawl a site and I'm relying on the default behavior where the same URL shouldn't be fetched twice (I never called AllowURLRevisit()). Most of the time this works fine, but I've…

Part of XiaomiMiMo/MiMo-V2.6-RL-oss.