Dynamic crawlers with RequestQueue often enqueue URLs that never get processed because of the…
Dynamic crawlers with RequestQueue often enqueue URLs that never get processed because of the…: a task in LegoFlow-SWE (Harbor dataset). The problem: When calling enqueueLinks() or addRequests() on a crawler instance, links continue to be enqueued even when the RequestQueue has already reached the…
The task
The problem: When calling `enqueueLinks()` or `addRequests()` on a crawler instance, links continue to be enqueued even when the `RequestQueue` has already reached the `maxRequestsPerCrawl` limit. These excess enqueued requests will never be processed, but they still incur the cost of writing to the RequestQueue.
Part of Lego-X/LegoFlow-SWE.