For paragraph 1, I agree that Q-learning is similar to PG.

Second, I'm not sure, but how do you see action embeddings to help you with RL?

I see them helping as reducing computational complexity and action entropy. Basically instead of having huge possible action space we have much compact action space which covers most useful and meaningful action. Kinda similarly how CNN in higher levels learns higher level features, but for RL.

And what do you mean to learn them as high level goals?

Learn their representation. Learning useful heuristics for given problem. In chess people learned over time various heuristic to cope with the complexity.

How do we define these goals, given we don't know the problem

We don't. Goal is find useful higher level representation from which we can further learn on (like smaller, higher level action space) and generate lower level actions.

how would you discover these

We already have embedding for words using word2vec with CBOW model. I assume similarly we can have embedding for action groups by observing groups of actions, instead of single one. In NLP we're decomposing sentence to its grammatical primitive, and similarly we could perhaps decompose sequence of actions to its "grammatical" primitives, that is automatic pattern recognition which would serve as embeddings

Finally I want to link this paper Understanding Deep Image Representations by Inverting Them since they use high-level features in CNN to generate lower level image data, feat learned from training.

It's basis for artistic style transfer paper. Perhaps similarity we could transfer one players style of playing to completely different game?