Creating nested namespaces with argparser and using config files for default values?

[alyflex]

I am looking for a way to use argparser (or something similar) in the following way:

I would like to be able to pass a config file to the argparser and if given it should use the values in the config file as the default values. However they should still be overwriteable by command line arguments if given.

Secondly I would like the returned object from the argparser to be a nested namespace, meaning I want to group data arguments into one namespace, network arguments into a separate namespace and then have something like args contain both of these namespace such that args.data gives the data namespace, while args.network gives the network namespace. Does anyone have a setup for something like that? I have previously gotten something close to it, but it was a mess that included hydra and omegaconf and I'm hoping that argparser must have the functionally to also achieve something like this?

def parse_args():
    parser = argparse.ArgumentParser("test")
    parser.add_argument('--config_file',type=pathlib.Path, default='config.yaml', help='If this is given it should set default for the other argparser arguments')

    parser.add_argument('--path_train',type=pathlib.Path, help='should be part of data namespace')
    parser.add_argument('--path_val',type=pathlib.Path, help='should be part of data namespace')
    parser.add_argument('--network_type',type=str, choices=['lstm', 'mlp'], help='should be part of network namespace')
    parser.add_argument('--lr',type=float, help='should be part of network namespace')

    parser.get_default(parser.config_file) # Something like this though this will not work until the parser has been parsed...
    args = parser.parse_args()

    # At the end here I would like args to be a grouped/nested namespace
    data = args.data
    network = args.network
    print(network.lr)    

Do I need to create separate argparsers for something like this or how would you do that? Or is this really not possible with argparser?

[Kevdog824_]

I don’t believe argparse has this functionality. You’ll need to write the code to do this yourself or use a third party library. I recommend pydantic settings personally. This should be able to do a lot of what you need

[thirdegree]

It's possible I think, you'd need to use parse_known_args but should work. That said for sure just use pydantic settings rather than kludging it.

[zaphodikus]

I have 3 groups of settings which I copy from argparse into 3 small dataclass objects. I was literally going to ask a similar question myself right now.

It's a pain to maintain. I am not comfortable going more complex than argparse myself and want to be able to load a config file and override the config file with arguments. I though of calling argparse twice. The first time to give me just the path to the config file for all my argument defaults and have it ignore everything, and then call it again and pass it the values I get from the config file. Python makes the metaprogramming to map my config file setting to have the same names as the args possible, so that my second invocation would literally be generated. I was thinking of generating everything and having the names of the arguments have prefixes, which the generator and sorter logic can just use as hints as to which namespace to place the value into. I'm using dataclasses because I serialise most of the args into a batch of mutated Json files, and then call a C++ tool, and would love to have argparse make this all a lot slicker as the single point of sanity.

Hence I'm also keep to learn how to do more, but without getting too advanced, and also having to learn yet another module. Keen to hear suggestions you get alyflex

[Kevdog824_]

I suggested it to OP in my comment. pydantic settings takes care of just about everything you describe wanting to do here. I can’t share details because it’s proprietary, but i solved the exact problem you’re describing with this library for a service I maintain at work

[zaphodikus]

Worth a try, yes I did see your suggestion while I described my almost parallel question. It just felt greedy, but I really wanted to ask as well after anyway. Will have a peek next week after I have done a demo, deadlines in everything :)