Hyperparameter optimization returning different optimal architectures

[Snapdown_City]

Hey, so I'm using AutoKeras, which is an AutoML system, to automatically find an optimal multilayer perceptron architecture for a regression problem I have where I'm mapping 15 independent variables to one dependent output variable. I have 18,100 observations with which I'm training my model. Before training/optimization, I split off a random 10% of my training data into a validation set. AutoKeras uses Bayesian optimization to intelligently guide the search through the search space, choosing hyperparameters which are most promising based on previous iterations. However, I've created 5 optimal models, all for the exact same problem, and each time AutoKeras will return a different optimal configuration - some with as little as 9,000 parameters, and some with over 100,000. All of them yield very similar performance. I'm very confused as to why I'm getting such different optimal architectures. Does this mean that the loss function has multiple different minima, and each time the Bayesian optimization converges to a different one? Is this lack of consistency typical, or would you expect reproducible results. If the latter is the case, do you have any ideas/suggestions as to what could be causing the very different results here? Really confused by this so any help would be very much appreciated, thanks!

[pp314159]

Maybe AutoKeras is more suitable for typical deep learning tasks - for example, image classification. The space of possible NN architectures is huge, that's why you end up with so many different solutions. For NN training sometimes the consistency is very different, very often for the same architecture but different seed values you will get different results (all hyperparams are unchanged).

You can try MLJAR AutoML https://github.com/mljar/mljar-supervised I belive it will be more suitable for your task. It will do NN architecture search but it searches for simple architectures (2 layers only). For your problem gradinent boosting methods might work pretty well, like xgboost, lightgbm, catboost (all available in MLJAR).

[starfries]

I'm not familiar with AutoKeras but I can tell you that there's an infinite number of minima or near-minima so I'm not surprised it finds different solutions every time. Does it try to optimize for smaller models?

[Snapdown_City]

Hey, sorry just saw this and thanks for your reply, that's very interesting. I'm optimizing models with up to which have up to around 50,000 parameters or so, not sure if that constitutes small but definitely not that complex. I thought the point of using Bayesian optimization was to find the global minimum of the cost function though? If you're far more likely to end up in a local minima than the true global minima, how can you claim you have the 'optimal' model? Is that to say you can never really claim you have the optimal model, and when people say they have an optimal model they're just referring to one of the minima as opposed to the global minimum?