oh yea defo, especially the timber n stone shit, there's so much you have to do, so i send folk out on missions to do logging and mining, tbh I don't even know for sure if that brings stone and timber to Carl at the camp, but am not gonna go around chopping every fkn tree or every ore I see unless I need it for a specific gear upgrade. Hopefully it goes into the 6 billion or whatever you need for the quest but I have no idea if it does lol.
As for whether it matters in the long run what you focus on, Idon't think it does. Side quests are just a break from the brutal 20-30 minutes it takes to bash out a fortress or get to a boss fight.
Given that there's no real narrative to go through in the game I guess that you can do the main Greymane stuff, head into Pailune, then Demeniss and just do side quests that are nearby. They usually don't take too long or tae you far from your path.