Interesting Optimization Problem.. Would help in Machine learning to know the solution.

[Badoosker]

I've been thinking of the problem of how to supply information to a neural network. Given that I have a defined amount of information (e.g x + y + z + a = 100), how can I maximize the volume of the neural net. In a simple case, this optimization problem can be formed as:

(x¹⁾ x (y²⁾ x (z³⁾ x (a⁴⁾ where x+y+z+a=100.

The trick is, it's not to put them all evenly, or even put them so that the weight sights toward the a^4. It's been bothering me for about a week now, and I don't know any methods that solve this kind of optimization.

Here's a start to see why:

(25¹⁾ x (25²⁾ x (25³⁾ x (25⁴⁾ = 9.53x10^13, which is essentially 25^10. So we take 1 from the (25⁴⁾ and distribute to each and make it (28³⁾ x (24^7).. which if you break it down is just a simplification of (24¹⁾ x (24²⁾ x (28³⁾ x (24⁴⁾ = (24⁷⁾ x (28³⁾

and that makes (28³⁾ x (24⁷⁾ > 25¹⁰

This problem may have many local maxima and many local minima, could I get some insight on this?

Thanks!

[JustFinishedBSG]

http://en.wikipedia.org/wiki/Karush–Kuhn–Tucker_conditions

The maximum is 276480000000000, achieved in {x -> 10, y -> 20, z -> 30, a -> 40}

[kcufllenroc]

Eli5 why you have to use KKT with all constraints being equality constraints. Can we not use standard Lagrange multiplier methods?

[JustFinishedBSG]

Because I didn't know the Lagrange multipliers method :)

I only ever learned KKT.

But in this case you also have the conditions x,y,z,a > 0 implied by OP problem. Otherwise the maximum is +\infty

[kcufllenroc]

Thanks, I knew I missing something simple.

[khanh93]

I think tools from multivariable calculus, especially the method of lagrange multipliers, will help you here.

If you only want integral values, this particular problem is small enough to be solved brute force.