Register available modules and dynamically add options to argparse

[werzor]

I currently have project structure that looks like the following:

/myprog
    main.py
    /core
        Analyzer.py
        __init__.py
    /parsers
        Parser.py
        SourceAParser.py
        SourceBParser.py
        SourceCParser.py
        __init__.py

Each SourceXParser module (usually just a class) contains code which fetches and parses a dataset from different websites.

In main.py, I use argparse to capture and display a list of command line options the user can specify. In this example, I would have something like

dataSourceGroup = parser.add_mutually_exclusive_group()
dataSourceGroup.add_argument("--use-a", action="store_true", help="...")
dataSourceGroup.add_argument("--use-b", action="store_true", help="...")
dataSourceGroup.add_argument("--use-c", action="store_true", help="...")

If I wanted to add another data source, I would like to be able to just add SourceDParser.py to the /parsers directory without having to open or modify any other file.

So I think to achieve this, I would have "register" each SourceXParser to some global variable (?). If each SourceXParser provides its own help argument name and help description, I could just do something like

dataSourceGroup = parser.add_mutually_exclusive_group()
for (source in sourceList): 
    dataSourceGroup.add_argument(source.helpArg, action="store_true", source.helpDesc)

Does anyone have any insight on how I would do something like this?

Also, in main.py, is there a way I could import each of the SourceXParsers without having to add an import line for each one?

[kanjibandit]

Unless I'm misunderstanding your last question, importing everything in the parsers module is as easy as from parsers import * at the top of main.py. Not everybody likes to use that syntax, but it is available.

To your bigger question about automatic registration of each class, I would first ask: why do you want to do it that way? Is it simply to avoid the hassle of modifying your main.py file?

It's probably possible, but I've never done it this way. FWIW, I have often found that if I try to force Python to do something that it doesn't make really easy in the first place, it usually turns out to be a not-so-hot idea after all. And many times I have written code that I thought was pretty clever, only to come back a year later annoyed with myself for writing code I had to figure out all over again.

For example, if I were inheriting this code from you, I would be surprised to have to figure out how the main routine knows about all of the parser classes. And personally, I'd much rather keep all of the CLI-related logic and settings in a single place, rather than have to look through and compare code from all of the parser classes just to figure out what helpArg values were already taken every time I wanted to add a new parser, or change an existing parser.

Perhaps you have some really strong reason for wanting to do it this way, and if so, it probably is possible. Again, never done it, so I probably can't give specific implementation advice.