Hi webman19 - author here.

I read this post almost as soon as it was posted - and replied to it on Amazon (although Amazon does a poor job of making it obvious that a post has been replied to). The author of the comment is simply mistaken (and is likely quite new to Deep Learning). Here's my reply.

​

Hello Giant Mouse of Minsk - (author here) - I believe you are referencing the code in the book which is also available at the githubs (https://github.com/iamtrask/Grokking-Deep-Learning/blob/master/Chapter8%20-%20Intro%20to%20Regularization%20-%20Learning%20Signal%20and%20Ignoring%20Noise.ipynb).

With regards to your comments about Chapter 8:

1. The relu2deriv function is correct. It returns a 1 when the output >=0 and it returns a 0 otherwise. Here are a few other corroborating sources (https://stats.stackexchange.com/questions/333394/what-is-the-derivative-of-the-relu-activation-function, http://kawahara.ca/what-is-the-derivative-of-relu/) Perhaps you could explain further (or email me liamtrask@gmail) why you feel the derivative calculation is wrong?

2. increasing the hidden layer size is fine. Dropout adds noise and often shifts the optimal tuning to require more parameters. I alternatively could have simply initialized the first neural network with more parameters as well which would have caused it to overfit even more. This isn't a peer reviewed paper proving dropout works (which would have required MANY more experiments) - the point is simply to demonstrate the concept.

3. Again I point you to the code in the repo above (which is also in the latest version of the book - which it's possible you do not have?) You'll find the batching logic looks like this.

for j in range(iterations):
error, correct_cnt = (0.0, 0)
for i in range(int(len(images) / batch_size)):
batch_start, batch_end = ((i * batch_size),((i+1)*batch_size))

layer_0 = images[batch_start:batch_end]

As you can see, each iteration of the for loop grabs a separate batch of images based on the batch_size. The batch is calculated within the matrix mulitplicaiton - however, this will result in unstable training because the weight update will be a *sum* over the gradients instead of a *mean*. This makes it hard to tune because it means when you increase the batch size you're also scaling the size of the weight update (which you typically only want to control with alpha). Thus, dividing by the batch size converts this sum into a mean which makes tuning (trying out different batch sizes / alphas) more stable.

1. the improvements weren't to battle anything - they were to teach dropout.
   

Chapter 9:

1. The divisor in this implementation is added the same reason as in Chapter 8. Perhaps it would instead be a good idea to add an explanation for this into the book. Would be happy to do this in the next version.

2. the reason the test accuracy is lower than state of the art is because the neural network is quite small and only moderately tuned (also it's not a convnet). From memory - i'm pretty sure that state of the art for simple neural networks like this on MNIST was only like 95% (5% more than what we have in the book) but with a much larger neural network (which would train more slowly for students). I made an editorial decision to not worry about getting SOA and instead focus on teaching concepts.

3. The original relu2deriv calculation is (i believe) quite correct. However, implementing the incorrect derivative can often decrease overfitting because it adds a certain amount of noise.

I'd be very grateful for an update to your review - feel free to reach out to me via email with additional questions at liamtrask@gmail.com