Duplicate URLs being fetched when redirects are involved ( AllowURLRevisit=false )

Duplicate URLs being fetched when redirects are involved ( AllowURLRevisit=false ): a task in MiMo-V2.6-RL-harbor-code: MiMo-V2.6-RL Code (Harbor) (Harbor dataset). I'm using colly to crawl a site and I'm relying on the default behavior where the same URL shouldn't be fetched twice (I never called…

The task

I'm using colly to crawl a site and I'm relying on the default behavior where the same URL shouldn't be fetched twice (I never called `AllowURLRevisit()`). Most of the time this works fine, but I've noticed two situations where the same page ends up being downloaded multiple times.

Part of FineEnvs/MiMo-V2.6-RL-harbor-code.