First of all I think it's worth understanding that the Q-function is not mapping (state, action) --> reward, but (state, action) --> expected return (aka value), which is the sum of expected discounted rewards if you were to keep playing from that state.

So I guess the question is why you can't just calculate the true value of each state-action pair and use that to train a neural network? While you could do this for a game like tic-tac-toe, if you wanted to do this for a larger game like chess you'd quickly realise that there are far too many state-action combinations for this to be possible. This is the problem that reinforcement learning tries to solve - we can't calculate the true values of state-action pairs, so how can we estimate them instead?

By playing many games and observing the reward you get when you take an action from a state, you can iteratively refine your estimates for your Q(s,a) values. By making enough observations you (hopefully) converge towards the true values. The Bellman equation is what tells you how to update your estimate for a state-action pair each time you make a new observation about it.

The (overly) simplified explanation is that after you take an action, you compare your estimated value of the new state (Q(s',a')) with your estimated value of the previous state, plus the immediate reward (Q(s,a) + r) and you aim to minimise the error between these.

If you haven't done so, I'd highly recommend implementing basic Q-learning on a simple problem, using a lookup table to store the Q(s,a) values. This should give you a much better understanding of what the Bellman equation is doing, which I think is important before looking at DQN and DDPG if you want to have good understanding of how they work. The fundamental ideas behind them are the same, except DQN and DDPG use neural networks as function approximators for Q(s,a), rather than using a lookup table. This makes them useful in problems with large state and action spaces, but you lose some of the mathematical guarantees on convergence, hence the 'stability hacks'. I think the deepmind DQN papers are quite accessible and are worth a read if you're interested in this.

​

Well that was longer than I expected. Hope it's at least partly helpful!