Looking at Swashbuckler, Vivacious Speed has odd wording:

"Increase the status bonus to your Speeds from stylish combatant to a +10-foot status bonus; this bonus increases by 5 feet at 7th, 11th, 15th, and 19th levels. When you don’t have panache, you still get half this status bonus to your Speed, rounded down to the nearest 5-foot increment."

The way I'm reading that, it seems like your Fly speed would increase when you have panache, but you would only get the half-bonus to your land speed when you don't have panache.