>>936260725
my bad I'm retarded so what I did was to scrape the urls for 150k images form thebarchives and sort them by their database ID, now while they download I will only see first unique and new files and those with the fault I've explained.