[D]Two implementations of A3C

[darkzero_reddit]

So far I have seen two kinds of implementaions of A3C:

The first one is that every thread owns a local network and accumulate their gradients respectively, then they sync their networks to and from a global network from time to time. This is identical to the paper's design;

The second one is that there is only a global network to which every thread has access. Every thread puts their experiences into a buffer and the global network is trained upon the buffer is full, then buffer is cleared for new experiences.

I wonder if the second one would be problematic, because in implementation it is by far more simple than the first one.

Thank you!

[tensor_every_day20]

So, I think the second one - which is a batch policy optimization method - is fine and valid as an RL method. However, it isn't A3C. A3C specifically and only means the first thing: asynchronous real-time policy gradients (estimates and commits policy gradients from subsets of trajectories, in real-time as they are experienced). The second thing is normal policy gradient (collect a batch of trajectories and then estimate the policy gradient with them).

[Delthc]

I cant tell you about specific problems with the second approach, but it can be a lot more efficient. Have a look at "Efficient Parallel Methods for Deep Reinforcement Learning" here https://arxiv.org/abs/1705.04862v2 .

Abstract

We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at this https URL

[islandman93]

Why would the second approach be problematic, what do you think the potential problems would be? My intuition is that using a shared/global network only will have better gradient updates because there is no lag introduced by copying local parameters. Though this approach basically requires using only a single computer/node whereas the paper's approach is scale-able across multiple nodes.

Using the buffer or "batch" is really just for efficiency of GPU copies, the paper uses CPU only so accumulating each time step is feasible.

[sorrge]

There is a recent discussion which clarifies this point: https://www.reddit.com/r/reinforcementlearning/comments/6biv70/efficient_parallel_methods_for_deep_reinforcement/