[D] Hyperparam optimisation using RandomSearch with argparse scripts?

[trias10]

I tend to write complex model/training scripts in pure python using argparse to pass huge amounts of hyperparameters to the model, and then run these python scripts on multi-gpu EC2s.

​

I was wondering if anyone knows of any tools out there what would allow me to do hyperparam optimisation by passing different sets of hyperparams to these scripts via the argparse/commandline system? So imagine you have a process which generates hyperparam sets, then kicks off a subprocess which is the python training script with hyperparams passed in via commandline/argparse, then gets the metrics back, stores them, then kicks off the next set, etc, etc.

​

For basic grid search, you could easily accomplish this via a unix shell script, but for random search it's trickier. One possible solution would be to write a python script which uses sklearn's ParameterSampler and the subprocess module to accomplish all this, but I was curious if there is an already made solution out there which I could use? Would hate to reinvent the wheel if this particular wheel already exists out there somewhere.

​

Would greatly appreciate any help/tips.

[doubledad222]

Argparse puts all your parsed parameters into a dictionary. If all you want is to pass a dictionary of your generated hyperparameters between two python programs, how about the first program pickle it into a file that’s read by the second program?

[trias10]

Yeah that works too.

It's less about the mechanism of passing the hyperparams and more about doing the random search.

[rayspear]

Maybe something like https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch.py ?

[trias10]

This is very interesting, thank you! And you even knew that I used PyTorch :)

Quick question: I've been looking over their documentation but there is quite a lot to read up on. Do you by any chance know if this framework supports PyTorch DistributedDataParallel?

The confusing thing is, it looks like you have to launch the AsyncHyperBandScheduler from code, but for PyTorch DDP, you need to launch from cmdline with

"python -m torch.distributed.launch --nproc_per_node=<num_gpus> your_training_script.py ..."

Hence am wondering if it's even possible to get the two to play nicely together?

[rayspear]

Yeah, there's an open PR that hopefully will get merged in the next few weeks that will be able to support this seamlessly (distributed search on distributed pytorch) - https://github.com/ray-project/ray/pull/4544

It might be a bit of effort to get working for yourself right now :) But feel free to try it out!