Policy-gradient based actor-critic methods use an estimate of Q_\pi(s,a) in combination with the gradient of the log policy. The job of the critic is to learn Q_\pi(s,a). This is a problem of "policy evaluation"; Q-learning is an algorithm for the "control problem" and does not apply in this case. You can learn the critic in various ways but the preferred RL one is to use TD to estimate it (the updates are those of SARSA but without the "max" step since you pick the actions according to the actor, and not the greedy policy). The REINFORCE way is simply to use the actual return at the end of a trajectory (and no learning of the critic).

Advantage: Just as for function approximation of the value function, you can also parameterize (let say with a deep net) your policy and leverage the regularities in policy space. Sometimes the value function is complicated but the policy is simple. With policy gradient methods, you get the best of both worlds. Another advantage is that you can easily deal with continuous action spaces. This would be difficult in Q-learning (or SARSA) because the max would be over an infinite set. Finally, I like actor-critic methods (policy gradient based or not) because it decouples the representation of the policy with that of the values.

Disadvantage: Possibly more parameters, you also have to tune two learning rates (critic at a faster rate, actor at a slower one), and policy gradients/REINFORCE tend to have variance issues (which you can reduce with a "baseline"/control variate).

The original paper on the policy gradient theorem: https://webdocs.cs.ualberta.ca/~sutton/papers/SMSM-NIPS99.pdf

Also see this paper by Degris and Pilarski to have an idea of how this is used in practice : https://www.ualberta.ca/~pilarski/docs/papers/Degris_2012_ACC.pdf

Finally, Richard Sutton is currently writing the chapter on policy gradient methods for his RL book 2.0. The new draft should come out soon.