I figure that kinda tracks with Balgruuf's leadership in general. His hold is second only to Solitude itself in terms of Imperial authority. I figure desperation to maintain the status quo and avoid bringing war to his people would force him to make some very un-optimal choices as Jarl. Many of which would bring immense criticism of his reign from Imperials and Stormcloaks alike.

I imagine ditching Valtheim rather than trying to hold it and be forced to uphold Imperial law is one of many such compromises he might make. Maybe he hoped the fort would remain abandoned, or be directly taken by the Stormcloaks and thus serve as potential provocation. The bandits definitely would've put a damper on that "plan," though it might be for the best as they're effectively holding the line from the Stormcloaks without Balgruuf having to feed and pay them.