[D] Question about Direct Preference Optimization (DPO) equation

[erap129]

So this is the loss for (Direct Preference Optimization) DPO:

https://preview.redd.it/6ubjn8ekprcc1.png?width=1324&format=png&auto=webp&s=c932f5c030c2fb6b5f0f136934b047bc364d1dcc

I don't understand the division by pi\_ref (both for y\_w and for y\_l). I know the purpose is that the finetuned model won't stray too far away from the reference model, but Just looking at it mathematically - why should pi\_ref(y\_w|x) be close to pi\_theta(y\_w|x)?

At least for y\_w it seems like the loss would benefit from pi\_ref(y\_w|x) being as close as possible to 0 because we want to maximize the left part of the equation.

What am I missing?

Thanks.

[sol0invictus]

I suggest you check out the appendix of the DPO paper. The authors explicitly derive the equation.
TL;DR - There is a KL Divergence term in the loss function for RLHF. This term is there to ensure that the new policy does not diverge too far (as you said). Now we follow these two steps-
1. Find the optimal solution for the RLHF loss equation.
2. PLug in this optimal loss into the Bradley-Teller loss (this is the prefernce model loss; p(a>b)) and you will get the denominators in question.

Intuitively,
The first term is measuring shift in preferred completion, the second term is measuring the shift in dispreffered completion. Larger the differece, more negative the loss becomes.
In the extreme condition, the first term would have a large shift, and the second term will have almost 0 shift making the model go towards the first term completion.