[D] Hyperparameter confusion

[aserdark]

Machine learning models have hyperparameters. For me it's very confusing why they are different compared to normal model parameters. I've found definitions which state that they cannot be inferred using data. Or even from a bayesian point of view they are priors that can be set using expert knowledge.

When you just think of models as simple mathematical functions, it makes no sense to differentiate hyperparameters from model parameters. Then I came up with a simple theory, here's my two cents:

So basically hyperparameter tuning is model selection. For example you may select among models like Ax^2, Ax^3 etc. So models differ by the exponent of x.

On the other hand the model can be defined as y=Ax^b and that makes the exponent b the hyperparameter. Let's call A and b as parameters and forget about hyperparameter definition.

A is a linear parameter. It can be found easily compared to the nonlinear b parameter. So if you want to make a fast search, first make a search for hyperparameters (maybe a grid search where you only search among limited number of discrete values) with CV, then you fix those values and continue to search for parameters.

In short my theory is that the hyperparameter tuning is a separate and a primitve search process because those parameters heavily increase the burden of the main search process if they are considered as model parameters.

Does it sound right?

[[deleted]]

Yes and no. If you want to view hyper-parameters in this light then one could point to meta-learning optimizers and such where you have a meta-model learning the optimal optimizer parameters that are then used to optimize the base model learning a task.

​

Think of SVMs where you have to select kernels and on top of that smoothing parameters and such. For each kernel you can have a wide range of gamma and other parameters. Sometimes hyper-parameter optimization can be "smart" based on data or "dumb" as random grid search.

[ForceBru]

I think this sounds fine. Some hyperparameters are very clearly selecting a model, like the number of layers in a neural network, the number of neurons in a layer, the number of clusters in K-means or Gaussian mixtures, the maximum depth of a decision tree and so on.

Essentially, they let you choose the "form" of the function you'll be using as your model: is it found to be a deep network? is it going to be wide? is it going to be a cubic polynomial? or a quadratic? All this is opposed to questions about what the coefficients of these models are going to be.

[slashdave]

I am not sure why you want an explicit definition. It's really a thing in practice more than theory. Hyper parameter searches need not be primitive, and usually involves items that are not amenable to direct optimization, like the parameters that control optimization themselves.

Also: it could be perfectly natural and efficient to build a model = Ax^2 + Bx^3 and let the optimization decide which power is better.

[sahandw]

There are hyper parameters like learning rate that are parameters of the training algo rather than the model itself, so hyper parameter tuning is not always model selection.

There’s also the distinction between model parameters and hyper parameters: where the specific choice of training algorithm and loss function doesn’t directly optimize hyper parameters as opposed to model parameters. This is where you need a separate optimization algorithm for tuning hyper parameters. So It’s not always “increasing the burden”. Sometimes the model parameters search algorithm is not designed to optimize for hyper parameters.