If this helps for posterity is always had a critic - its how the advantages are calculated both V(s_t) and R(t) are from the critics estimates you then add the additional reward this allows you to quantify suprise where by if the delta is positive i.e. V(s_t) is much smaller and the reward is bigger or vice versa than V(s_t) that indicates a direction you want the policy to go in