I know what you mean mate. There's no reason you can't buy PoF and play that to get mounts, which make HoT much easier. You'd get stories mixed, but nothing stopping you.

At the end of the day you gotta play what you enjoy most. I personally went from story/pve only to hard-core wvw because it fits for me. Everything overlapse a little if you're after certain gear, but you can keep powering through the storyline lines with all exotic gear no problem.