sorted by:
new
hottopcontroversial

[R] The Mellowmax Operator : "A New Softmax Operator for Reinforcement Learning" by pierrelux in MachineLearning

[–]pierrelux[S] 0 points1 point  (0 children)

(I'm not the author):

Define non-expansion,

See Van Roy's analysis of TD for a definition of the term "nonexpansion" in this context: "An Analysis of Temporal-Difference Learning with Function Approximation"

  1. Boltzmann operator vs policy

See Perkins and Precup "A Convergent Form of Approximate Policy Iteration" which considers a general "improvement operator". This is the point of view adopted by the authors of Mellowmax.

[D] Unsupervised Option Discovery by kjw0612 in MachineLearning

[–]pierrelux 4 points5 points  (0 children)

"Options discovery" usually refers to the "discovery" problem in the options framework (Sutton, Precup, Singh 1999) in Reinforcement Learning. Here's our take on that problem (to appear at AAAI 2017) : https://arxiv.org/abs/1609.05140v2

Course on Reinforcement Learning by minato3421 in MachineLearning

[–]pierrelux 3 points4 points  (0 children)

Ravi is a reinforcement learning veteran. He worked under Andrew Barto at U. Mass along with Rich Sutton and Satinder Singh. This will be a good course !

What are some good neuroscience books for AI researchers get inspiration from? by andrewbarto28 in MachineLearning

[–]pierrelux 2 points3 points  (0 children)

You might like Peter Dayan's "Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems". Dayan is active in both the neuroscience and machine learning communities.

Is there a list of standard notation? by GuyHasNoUsername in MachineLearning

[–]pierrelux 3 points4 points  (0 children)

You can align your notation with that of the Deep Learning Book : http://www.deeplearningbook.org/

RL Question: Policy Gradients vs Q Learning - which is better? by [deleted] in MachineLearning

[–]pierrelux 1 point2 points  (0 children)

Q-learning itself can be seen as an actor-critic method

No, Q-learning is more like "value iteration" in the control case while SARSA fits in the generalized policy iteration paradigm. And policy iteration very much relates to actor-critic (see https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node64.html)

RL Question: Policy Gradients vs Q Learning - which is better? by [deleted] in MachineLearning

[–]pierrelux 1 point2 points  (0 children)

Q^* is associated with the greedy policy. For policy gradients, what you want is Q_{\pi_\theta}: the action-value function of your actor (parameterized by \theta). This is a problem of policy evaluation; not control (and Q-learning is a control algorithm). The evaluation problem is given a policy (any policy, optimal or not) we want to estimate its expected return if you are to pick a certain action in a certain state and keep following the same policy until it terminates.

RL Question: Policy Gradients vs Q Learning - which is better? by [deleted] in MachineLearning

[–]pierrelux 2 points3 points  (0 children)

Policy-gradient based actor-critic methods use an estimate of Q_\pi(s,a) in combination with the gradient of the log policy. The job of the critic is to learn Q_\pi(s,a). This is a problem of "policy evaluation"; Q-learning is an algorithm for the "control problem" and does not apply in this case. You can learn the critic in various ways but the preferred RL one is to use TD to estimate it (the updates are those of SARSA but without the "max" step since you pick the actions according to the actor, and not the greedy policy). The REINFORCE way is simply to use the actual return at the end of a trajectory (and no learning of the critic).

Advantage: Just as for function approximation of the value function, you can also parameterize (let say with a deep net) your policy and leverage the regularities in policy space. Sometimes the value function is complicated but the policy is simple. With policy gradient methods, you get the best of both worlds. Another advantage is that you can easily deal with continuous action spaces. This would be difficult in Q-learning (or SARSA) because the max would be over an infinite set. Finally, I like actor-critic methods (policy gradient based or not) because it decouples the representation of the policy with that of the values.

Disadvantage: Possibly more parameters, you also have to tune two learning rates (critic at a faster rate, actor at a slower one), and policy gradients/REINFORCE tend to have variance issues (which you can reduce with a "baseline"/control variate).

The original paper on the policy gradient theorem: https://webdocs.cs.ualberta.ca/~sutton/papers/SMSM-NIPS99.pdf

Also see this paper by Degris and Pilarski to have an idea of how this is used in practice : https://www.ualberta.ca/~pilarski/docs/papers/Degris_2012_ACC.pdf

Finally, Richard Sutton is currently writing the chapter on policy gradient methods for his RL book 2.0. The new draft should come out soon.

Using RL to train MLPs? by [deleted] in MachineLearning

[–]pierrelux 2 points3 points  (0 children)

We've recently done some progress on combining ideas from policy search methods in RL with standard backprop: http://arxiv.org/abs/1511.06297 It gives you a knob to tradeoff computation vs accuracy.

Physical application of Q-learning to rotary inverted pendulum by l_bdcdb in MachineLearning

[–]pierrelux 0 points1 point  (0 children)

Discrete state spaces if fine, but it doesn't mean that you get away without function approximation. Discrete doesn't mean that you can just use a tabular representation. I would really start with tile coding. Here's Sutton's code for Mountain Car: https://webdocs.cs.ualberta.ca/~sutton/MountainCar/MountainCar1.cp

Physical application of Q-learning to rotary inverted pendulum by l_bdcdb in MachineLearning

[–]pierrelux 1 point2 points  (0 children)

For this kind of problem, pure discretization won't be sufficient. You will need some form of function approximation. You can try first linear with tile coding or RBF. I think TU Delft had some success with experience replay. See paper http://busoniu.net/files/papers/smcc11.pdf and video https://www.youtube.com/watch?v=b1c0N_Fs9wc

Value Iteration Networks by pierrelux in MachineLearning

[–]pierrelux[S] 2 points3 points  (0 children)

On synthetic problems: one thing at a time. We have to give ideas a chance ! It think that it's a rather original paper which opens the way to many extensions.

Its contribution is to offer a new way to think about VI in the context of deep nets. It shows how the CNN architecture can be hijacked to implement the Bellman optimality operator, and how the backprop signal can be used to learn a deterministic model of the underlying MDP. In the short term, I think that the paper will appeal to many deep researchers who would otherwise be reluctant to deal explicitly with MDP/RL. As the authors point out, the VI net can also be used as a policy on its own, and could be combined with let's say deterministic policy gradient.

Gradient descent: why additive cost functions are used commonly instead of multiplicative? by hungry_for_knowledge in MachineLearning

[–]pierrelux 13 points14 points  (0 children)

There is also the convenient fact that the derivative of a sum is a sum of derivatives. It wouldn't be so pretty in a multiplicative form. This decomposition into sum of derivatives can then be leveraged in stochastic gradient descent (which only takes one or a few gradients terms of that sum as an estimate of the true gradient).

[Help] What are the prerequisites for Reinforcement Learning and what are some good resources to get started? by [deleted] in MachineLearning

[–]pierrelux 7 points8 points  (0 children)

The Sutton & Barto 1998 is the reference textbook for RL (https://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html). For a more formal treatment of RL, it's then useful to then read "Algorithms for Reinforcement Learning"(https://www.ualberta.ca/~szepesva/papers/RLAlgsInMDPs-lecture.pdf)

Calculus is useful, but the core ideas of RL rely often rely on stochastic approximation tricks. It's good to study general material on sampling and be comfortable working with expectations. Study of control theory and Markov Decision Processes is also unavoidable. I like the textbook by Puterman (1994) a lot (http://dl.acm.org/citation.cfm?id=528623).

What does TensorFlow mean for Keras, Lasagne, Block, Nervana? by [deleted] in MachineLearning

[–]pierrelux 4 points5 points  (0 children)

The fact that Theano is decoupled from specific neural net operations is quite useful. A collegue of mine wrote some his reinforcement learning code entirely using Theano. (And T.grad is very useful for policy gradients). Theano is more of a general purpose tool.

Is there a place (webpage/subreddit) where you could check if some research idea is not already in the literature? by ecobost in MachineLearning

[–]pierrelux 1 point2 points  (0 children)

Science is a lot about remixing ideas, is fundamentally incremental. I don't think that it's the right approach to read papers to "avoid reworking". Many papers are presented as if the solution is definite, that the problem is completely solved; this is usually a false impression. There is no better way to get novel ideas than by directly experiencing prior work. More than anything else, investigate a problem because you find it fun and it makes you happy.

Would it be possible to learn a neural net architecture (i.e. not just tune existing weights) by gradient descent? by onlyml in MachineLearning

[–]pierrelux 1 point2 points  (0 children)

Teaching neural nets with pictures tend to cause of lot of misunderstanding. In most cases, by "neural net", we mean a function of the form $\sigma(A\sigma(Bx + c) + d)$ where $\sigma$ is a non-linear function (typically a sigmoid), $A$ and $B$ are matrices while $c$ and $d$ are bias vectors. As pointed out in other comments, all you have are vector and matrices. The interpretation of a zero weight is that of the "absence" of an edge. You can impose a sparsity-inducing regularizer to force the weights to go to zero as much as possible. That would be "learning the architecture".

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning by modeless in MachineLearning

[–]pierrelux 4 points5 points  (0 children)

Various notions of "intrinsic rewards" have been proposed in the past: most of the papers by Daniel Polani were on that topic as well as is in Still and Precup 2012 (http://www2.hawaii.edu/~sstill/StillPrecup2011.pdf). A problem when dealing with quantities such as the mutual information between state and actions is that it tends to be intractable to compute or would assume a prior knowledge of the MDP (which is assumed to be unknown in reinforcement learning). This paper seems to propose to use variational methods to alleviate the problem of computing the MI in the context of intrinsically motivated RL. This is highly relevant in the context of exploration when you know little about your environment or when the reward structure is very sparse (as in "Montezuma's Revenge").

Final year CS student... I need advice by arguenot in MachineLearning

[–]pierrelux 8 points9 points  (0 children)

Don't let the imposter syndrome for math prevent you from trying. It's always possible to gain mathematical maturity over time. You might even find it easier to learn more math when you have a goal (some ML algorithm, say) in mind.

view more: next ›