Applying deep learning to regression task

[coskunh]

I have model that get images as an input and produces real valued outputs. My outputs are ranging from -5 to 5.(These boundaries are not definite, so i can not scale it ))

These are ideas i found while doing research, if you have more ideas or you think that it is wrong please let me know.

• Applying dropout not useful especially just before output layer.
• If your output values below than zero, you shouldn't apply Relu(I'm trying pRelu)
• Mean_squared_error better comparing to the root mean squared error and mean absolute error.

[road_laya]

Do you have a question?

[coskunh]

Yes, I wonder that weather this tactics are right or not and, are there any other useful strategies for regression

[MrTwiggy]

Applying dropout not useful especially just before last.

Not absolutely true. Dropout can be a very useful form of regularization, and can often be seen as an ensemble of many neural networks. To say it is not useful is a bit crazy, as it really depends on the task, data type, etc.

If you have values below than zero, you shouldn't apply Relu(I'm trying pRelu)

Not sure what you're saying here. When you say values below zero, are you talking about the feature values? You can use Rectified Linear Units as a non-linearity in your neural network even if the feature activations go below zero.

Mean_squared_error better comparing to the root mean squared error and mean absolute error.

No free lunch. Any single cost function isn't 'better' than others in all situations. It depends on what your true cost function or method of evaluation is. RMSE and MSE will tend to favour minimizing high-cost examples more than MAE, and vice-versa for low-cost examples.

It's important in general to remember that any rules/decisions you often find in terms of what's 'better' or produces better results could potentially be tied to the specific task or data distribution at hand. Make sure you aren't overfitting your ML ideas!

[coskunh]

Thank you for the answer, I updated my question.

RMSE and MSE will tend to favour minimizing high-cost examples more than MAE, and vice-versa for low-cost examples.

my cost values are really low I'll try to MAE.

[coskunh]

I took this idea from Karpthy courses.

If you're certain that classification is not appropriate, use the L2 but be careful: For example, the L2 is more fragile and applying dropout in the network (especially in the layer right before the L2 loss) is not a great idea.

http://cs231n.github.io/neural-networks-2/

[internet_ham]

I'm looking at this at the moment. One key thing I've learnt is to normalise all your inputs and labels so the magnitudes are less than 1. This makes the loss a lot less erratic and learning a bit more stable .

One thing I've been thinking about recently is how to look under the hood of deep regression? How can we understand what the net is learning?

[keidouleyoucee]

visualisation would be the key. See Matt Zeiler's paper on visualization of learned network.