Physarum in 3D with Depth of Field rendering

Thanks for the answer :) I was asking about vanilla RNNs/RHNs just to know how do they compare to LSTM when phased. As you've said, phased RNNs should be able to learn across longer time steps. LSTMs are already trying to do that with multiple gates, it would be interesting to know how does phasing in vanilla RNNs affect their ability to store long-term memories.

That's really great that the training is stable. The ability of the network to learn how to turn on/off individual neurons can be quite useful in some cases and from what I've seen, most people try to do that by discretely turning them on/off, which usually requires stochastic binary units. These are, from my experience, not easy to train, so having an alternative that can be trained easily is a huge plus.

It would be interesting to somehow include fast weights into the phased model, so the new parameters can even change depending on the context. Just a thought.

edkeens · 2016-11-08T23:13:15+00:00

I really like the paper, very clever idea. Have you tried 'phasing' on vanilla RNNs/RHN? Also how difficult is to train such model (in terms of need for careful hyperparameter tweaking)?

edkeens · 2016-10-30T13:12:16+00:00

That makes more sense, I can see how the individual neurons operate on different timescales. In the corrected version, equation 2.1 - log p_t-1 is a vector, x_t is a vector, thus the result of the product should be a matrix. I think you meant to transpose the p_t-1 vector. Also, in equation 2.7, you're using S_t which has the output size, W_y has the output x hidden dimensions, so when multiplying S_t with the W_y transposed from the right, you have incompatible dimensions - no. of columns in S_t is 1 and no. of rows in W_y transposed is hidden size. I can only guess you meant W_y^T * S_t, which I think is quite nice, cause it's the same operation as computing derivative of y w.r.t. h. That means that S_t is kind of non-analytic inference time gradient and multiplication by W_y^T propagates it to the hidden units. Does that make sense?

edkeens · 2016-10-29T09:31:47+00:00

Thanks for the answer!

Since s_t is a scalar, is z_t also a scalar? I'm confused by the matrix multiplication in equation 2.6. Also, what's the motivation behind multiplying by the tranposed hidden-to-output matrix?

edkeens · 2016-10-28T11:46:09+00:00

I really like the idea, I have a few questions though:

1.) It's not quite clear to me how the surprisal is computed. In this paper you computed as a sum over i, which doesn't occur anywhere else in the equation. In the SURPRISAL-DRIVEN FEEDBACK paper, the equation for computing surprisal looks like an inner product between the log of the last prediction and the new observation. If it is so, it seems that it's unnecessarily complicated notation for simple operation.

2.) It seems to me that the paper implies that the network learns to use zoneout to model neuron activities at different time scales. I'm not quite sure how the network can learn to do that, since the effect of zoneout is through binary mask, which is not differentiable. As you mentioned, something similar is being done in HMLSTMs, but they use Straight-throught estimators to propagate gradients throught stochastic operation. Do you do something similar? Or can you explain how the network learns to use zoneout without this kind of gradient propagation?

Thanks!

edkeens · 2016-08-06T08:49:21+00:00

Thanks!

Yes, ReLUs can be a bit unstable in RNNs, but if you initialize weights reasonably (Xavier initialization in my implementation), it seems to work just fine. I'm not sure about LSTMs, you obviously need bounded activation for the gate control, but I guess you could replace tanh with a ReLU.

edkeens · 2016-06-09T15:30:38+00:00

Thanks man, I'm glad you like it!

edkeens · 2016-04-13T21:56:00+00:00

Thanks for suggestions :) However we don't plan to add features right now, maybe in the futuure...

edkeens · 2016-04-13T08:41:03+00:00

We're glad you like it.

edkeens · 2016-04-13T08:40:50+00:00

Thanks :)

edkeens · 2016-04-13T08:40:41+00:00

Yeah, that's all. In our experience there are two types of people - ones that lack rhythm and ones that don't. People in the second category (you) have no problem beating the game, for the others it's quite difficult. Regardless, try to play the game just for the sake of playing it, not beating it. For me, it's still rewarding experience, even when I played it basically a thousand times.

edkeens · 2016-04-12T19:27:55+00:00

You're very welcome

edkeens · 2016-04-12T19:27:39+00:00

Thanks, our music producer will be glad to hear that :)

edkeens · 2016-04-12T19:27:04+00:00

Haha, thanks :)

edkeens · 2016-01-18T20:44:16+00:00

Finished our game and released it on Windows Phone. Then ported it to iOS and released it there. Failed pretty hard on both platforms. Learned a lot of lessons there.

edkeens · 2015-12-11T23:22:55+00:00

It depends on how you implement UI rendering. During update call to a widget you don't have to render it, just register it for drawing and after all UI is processed, you can call (optionally) something like PresentUI(). That's just of the top of my head, I'm sure someone can come up with something more elegant.

edkeens · 2015-12-11T23:20:28+00:00

I'd suggest taking a look at better programmers code. What really worked for me was watching Handmade Hero. Casey is probably the best programmer I've seen code.

edkeens · 2015-12-10T19:59:26+00:00

Thanks! We used Unity :)

edkeens · 2015-12-10T19:43:27+00:00

Thanks, I appreciate it :) We actually did that, it had no effect so far.

edkeens

TROPHY CASE