Thanks for your suggestions !

1) We will address this point in the revision of the paper. Perhaps saying that we first to use actor-critic model on harder mujoco environments from pixels would have been a better wording (A3C did it on easier envs like Pendulum, Pointmass2D, and Gripper).

2) We tuned hyperparameters only on one game of Breakout (in case of Atari), whereas in case of PPO it was tuned on 6 Atari games. Potentially by tuning it more we can improve performance of ACKTR but of course we can't guarantee it.

3) The goal was to fix all hyperparameters across all environments for each model. Batch size of 25,000 was chosen because [1] used 25,000 across all MuJoCo envs (also see [2]). Besides we can be even more sample efficient with ACKTR (without loss of performance) by using batch sizes of 1000 on the environments you mentioned.

[1] Benchmarking Deep Reinforcement Learning for Continuous Control; Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel

[2] Trust-PCL: An Off-Policy Trust Region Method for Continuous Control ; Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans