Feature Request: Seamlessly combine RequestList and RequestQueue to avoid duplicate URL processing
Feature Request: Seamlessly combine RequestList and RequestQueue to avoid duplicate URL processing: a task in LegoFlow-SWE (Harbor dataset). When a crawler is configured with both a RequestList and a RequestQueue , requests are often processed twice because the list and queue operate…
The task
When a crawler is configured with both a `RequestList` and a `RequestQueue`, requests are often processed twice because the list and queue operate independently. Users must currently write custom orchestration logic to ensure URLs are not revisited. The crawler has some internal handling for this “tandem” scenario,…
Part of Lego-X/LegoFlow-SWE.