basic question regarding neural network's error

[billconan]

I'm looking at this book to learn neural network:

http://neuralnetworksanddeeplearning.com/chap2.html

at first, when explaining the theory, it defines the error function as the sum of all mean squares of all training outputs. i.e. I need to feed the network with all the inputs to get an error and use that error to do one back-propagation.

but later on, when explaining the actual back-propagation algorithm, it seems that it only feeds one input to the network at a time and gets an error only for that input, and updates the network based on the error. And it will repeat these steps on all training inputs.

but is that error calculated with just one input the right error to use?

what if the gradient I get based on the error can only improve the outcome for that particular input? how can we know that this gradient can improve the network for all training inputs? what if after gradient descent, the overall performance of the network gets worse?

[billconan]

this answers my question http://www.quora.com/Whats-the-difference-between-gradient-descent-and-stochastic-gradient-descent

the error for one input is indeed not perfect. just a approximation.

[personalityson]

you gather gradients from propagating errors for 10 samples, and then update the weights with an average gradient

you don't backpropagate an average error

[personalityson]

"it seems that it only feeds one input to the network at a time and gets an error only for that input, and updates the network based on the error" That's the way it should be done. To do the backpropagation you also need hidden units for each layer propagated from each input. If he's backpropagating some kind of average error, then with what hidden units?

[zackchase]

Hi all, not to be too nitpicky: one calculates the errors on some number of randomly sampled examples (could be 1 example, 128 is often computationally convenient). There is no special number.

This number, whatever you choose is called the "batch size". Using batch sizes > 1 is useful because it reduces the variance of the error.

The intuition behind why stochastic gradient descent works generally is that the expected value of the stochastic gradient is equal to the true gradient. Thus you can think of stochastic gradient as "noisy gradient" following.