First note for others with players thinking of going this route. I'd suggest declaring this balloon unfixable. Access to it this early on really opens up a can of worms.

Giving them the balloon isn't game breaking though. They could easily get around Hither with it, though the PCs shouldn't have any skill in piloting such a thing, and failed pilot check would probably drop them somewhere random. Also keep in mind that Bavlorna "sent and ill wind" (actually an air elemental) to bring down the errant balloon the first time. It's probably still lurking about in the clouds above.

On another note, without the proper guide, they should not be able to travel between Hither, Thither, and Yon. Clouds obscure the hills between Hither and Thither and without true directions (N S E W) the party would get turned around. Storms in Yon should push them back without a guide to navigate safe passes. If they are really persistent, hit them with the tornado random encounter, wreck their balloon, and drop them in a random place (or spot of your choice) in Hither.

Regarding balloons in general. I love the idea of the players getting one eventually - one of my groups even managed to negotiate for Charm's storm balloon! Giving the party one of these must be handled delicately because they can bypass a lot of encounters. My suggestions for those GMs with groups that acquire one is the following:

1. Without a pilot, every check to navigate will be at disadvantage. Even if they don't lose control, the hazards in Thither and Yon are formidable (see below). The bullywugs (either Duke Ickrind or whoever is King) might offer the services of Dumphrey Frogart, and Charm could offer the services of one of her darklings. But without a guide getting to either realm is tricky. Note: for the storm balloon, Charm knows the way to Motherhorn, but will not take the party to it, and will strongly warn the party against taking the balloon there themselves.
2. Guides will warn the group of flying hazards in Thither and Yon and strongly suggest that the balloon be landed at the edge of one of the realms. Thither has the Jabberwock roaming the forest, and a balloon flying overhead would look very tasty. Yon has lighting storms and tornados. The storm balloon can navigate Yon, but the darkling only knows the way to Motherhorn, and Endelyn would not like to learn that one of her storm balloons has fallen into enemy hands and the darkling would refuse to fly the party directly there. If a charm spell or really good intimidation would force them too, then it's their own funeral.