Dir Scan: Add per-directory download client filtering, fix path matching, and reject duplicate directory paths

Dir Scan: Add per-directory download client filtering, fix path matching, and reject duplicate directory paths: a task in LegoFlow-SWE (Harbor dataset). What the agent is asked, how it is graded, what it runs in, and its files.