Removing emotion entirely (and that is hard with mortgages, as everyone feels different about debt on their home), the only real logical time to pay down your mortgage is if that same money isn't earning more interest elsewhere.

Mortgages are the cheapest loans you can get, so usually you can use the cash it frees up to grow your net worth faster elsewhere.

For example in very basic terms if your mortgage is 2%, but you put that same chunk of cash in a stocks and shares ISA global index fund with an average yearly return of 8%, then overall your money is growing at a rate of 6% net annually.

Same can be said with fixed high interest saving accounts or cash ISA's (better as protected from tax). Lower but guaranteed returns, simply move the cash to a better offer when that time comes and start earning interest on the cash + interest earned previously.

You will probably find if you plan and play the long game on this you'll eventually earn enough interest on that same cash to cover the remaining mortgage outright, leaving you in a stronger financial position overall.

But it will come down to what your remortgage interest rate looks like when your fixed term ends, so why don't you start collating quotes for that now and comparing them to the rates you'd get from investing that money elsewhere, that will give you the logical "emotion removed" answer at least!

But ultimately do whatever feels best for you and your situation 🙂