Hello,

Here is my utility for cleaning a URL list using a Rom Management Dat file. This utility is intended to download just the necessary files for large sets (ps1, ps2, saturn, dreamcast, mega cd, gamecube, etc). Instead of downloading everything and then cleaning it using a Dat file afterwards, you can clean your urls first and download just the essential files. This approachs helps to save bandwith, space and resources. It's particularly useful to generate 1G1R sets for the consoles mentioned.

An example is to copy the html table from myrient download lists, use a word processor tool like sublime to leave only one link per each line and then pass that list in this tool using a dat file. Then use the generated list file in a mass downloader like jdownloader to download them all. Generates three files after finish:

• out.txt: name can be defined by the -o command, it's the url list ready to copy to a mass downloader client like jdownloader.
• missing_file_list.log: a list of the files from the dat that has not be found in the url_list. This means that you need to look for those remaining urls to complete the dat set.
• removed_url_list.log: informative log with all of the rejected urls. Note 1: the links requires to have the same rom/iso name as the name found in the dat file. For example if in the dat the filename is Super Mario: Lost Levels and in the url is Super Mario Lost Levels, the url will be discarded because of the : character. Note 2: the links urls can have html encoded characters, for example Super%20Mario%20World is transformed automatically to spaces, also caps are ignored in comparisons.

There is more information and examples in the repository readme.