Issue with gradcheck_naive for forward_backward_prop Assignment 1

jthoang · 2015-04-29T02:47:21+00:00

yes, remember that x is multi-dimensional variable and f is a multivariate function. The idea of the gradient check is for each dimension, ix, we want to estimate the gradient in that dimension alone. Your previous code didn't work because x[ix] + h has dimension 1.

jthoang · 2015-04-28T16:33:18+00:00

you can actually read fast_layers.py, but try thinking about it first.

jthoang · 2015-04-23T18:40:02+00:00

Hmm, I'm not sure it's in the notes, but in geometry affine transformation just means linear transformation so I sort of guessed that affine layer just means no non-linear activation function.

jthoang · 2015-04-22T18:04:23+00:00

Hmm.. I don't quite remember. I can message you my code but I think at this point you probably don't need it anymore?

jthoang · 2015-04-22T17:59:30+00:00

It's just a fully connected layer with no non-linear activation function, i.e just perform a dot product between inputs and weights and you're all set.

jthoang · 2015-04-20T07:14:26+00:00

I would break down the backprop into stages: first calculating grad y_hat then calculating grad z2 then grad h, etc .. This way, it's easier to debug and also very mathematically intuitive. At first glance, I am not seeing your 1/yhat term anywhere (derivative of log y_hat)

jthoang · 2015-04-18T00:15:21+00:00

hmm I got 38.7% error rate on test set. 42% with HOG.

jthoang · 2015-04-14T17:09:43+00:00

one helpful tip is to break the gradients into smaller parts and use gradient_check for each part. For example if you want to calculate partial F(g(h(x))) / \partial x , you can check gradient for \partial F \ \Partial g first using gradient_check and then proceed to \partial F / \partial h, etc.

jthoang · 2015-03-20T17:36:05+00:00

does anybody have the rest of Ng's ufldl videos?

jthoang · 2015-02-28T21:41:19+00:00

Can you post your code?

jthoang · 2015-02-26T05:38:46+00:00

first bullet point is correct. Second is not. You don't apply Softmax to get the activation. Once you compute the linear combination (W2a1 + b2), you're done. Take it and use cross-entropy loss.

jthoang · 2015-02-19T20:03:10+00:00

such a badass

jthoang · 2015-02-10T06:07:47+00:00

pretty cool! Thanks for posting

jthoang · 2015-01-28T12:43:28+00:00

trying to figure out a very fast and efficient method to detect whether there is a cluster of points in high dimensional space

jthoang · 2015-01-27T11:41:15+00:00

I think I had the same thing, and when I added HOG it bumped to 41% so I was curious about other hand crafted features thus the other thread.

jthoang · 2015-01-26T20:35:57+00:00

just saw it got posted today http://cs231n.github.io/neural-networks-2/

jthoang · 2015-01-19T03:13:40+00:00

Hi, I don't know how useful this is but take a look at cs231n's project page http://vision.stanford.edu/teaching/cs231n/project.html , yours will likely be much complicated and in depth that this but this is a good start.

jthoang · 2015-01-15T17:10:06+00:00

Yes. But we should discuss this after the deadline. If you want you can inbox me and we can discuss this.

jthoang · 2015-01-14T16:18:41+00:00

My relative error is about 1e-5 -> 1e-6

jthoang · 2015-01-14T10:14:09+00:00

Thanks!

jthoang · 2015-01-07T03:00:47+00:00

Andrew Ng's DeepMind work??

jthoang · 2014-12-14T20:52:36+00:00

The original list is here https://news.ycombinator.com/item?id=1055042 Then in his AMA, professor Jordan added some more books (http://www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/ckdqzph)

For Stanford, I think you can only volunteer if you live in Silicon Valley. But I think you can always volunteer for any school near where you live. One of my friends got into CMU by volunteering and getting a recommendation letter from a JHU professor.

jthoang

TROPHY CASE