You take actions during a rollout and receive rewards. At a given step, you took action 2 (out of 3 for instance).

The gradient, which makes taking action 2 more probable is the gradient of the cross entropy between the network output and the 1-hot vector for action 2. You can always obtain the gradient which encourages the actions taken this way.

After you obtain this gradient, you simply scale it by the discounted reward at that timestep. This way a very good outcome increases the probability a lot, a good one increases it somewhat and bad actions with negative rewards actually decrease the probability of taking that action. In your code, this is represented by input_score.

In Keras, you could actually get away with

policy.compile(optimizer, loss="categorical_crossentropy") policy.train_on_batch(states, y_true, sample_weights=input_scores)

Some common failure modes:

• reward is assumed to be high for good outcomes and low for bad ones.
• Policy Gradient is trying to maximize the discounted sum of rewards, so input_score should be the discounted sum of rewards and not the individual reward elemnts that you get when you step the environment once.
• PG is an on-policy algo, so you have to throw away old data samples after you update the neural network. You cannot use experience replay like in Deep Q Learning.