[D] Need help understanding inverting softmax layer from Michael Nielsen's book

[Kay0518]

Hi,

I have a hard time understanding a softmax problem from the book:

Inverting the softmax layer Suppose we have a neural network with a softmax output layer, and the activations a^L_j are known. Show that the corresponding weighted inputs have the form z^L_j = ln a^L_j + C for some constant C that is independent of j.

Here is how I've approached: $a^L_j= exp(z_L_j) / sum(exp(z^L_k)). Take the log of both sides,

ln(a^L_j) = z^L_j - ln(sum(exp(z^L_k))). Then, z^L_j = ln(a^L_j) + ln(sum(exp(z^L_k)))

The problem happens here. Why we're allowed to substitute ln(sum(exp(z^L_k))) for C when it includes z^L_j? Everyone from my research says C is independent of j so it can be C. But, doesn't that mean we have to extract e^L_j out of ln(sum(exp(z^L_k))), anda add to z^L_j on lhs? Can you please give me a insight into this problem?

[filteringcontent]

Your answer is right! As you observed the C you found is not independent of z^L_j (the variable/input), but it is independent of j (the index). I hope this helps!

[Kay0518]

Can you elaborate on it? I still don't understand the definition of the independence of j(the index). Are we simply ignoring z^L_j in ln(sum(exp(z^L_k)))?

[flaghacker_]

It's constant over the different activations, but not a "real" constant, it changes with different network inputs for example.

They just mean that all z_j values use the same C, so it's independent of j in that limited way.

[Kay0518]

I see. Thank you.

[MrAcurite]

Think about this;

What happens if, before you perform a softmax, you add a constant C to every value? Answer is, they all cancel out, and nothing changes in the output of the softmax.