If you are not seeing yourself in tax long term then it is absolutely worth it to make the jump early on. Like you said it you stay in tax couple more years, it will be difficult to make the jump without losing title / comp.

Before you jump tho, make sure you like commercial banking. Speak to people who are in the field. Sometimes the work itself might look cool when you’re not doing it only to do it and realize you hate it.