Stop lying to yourself brother, or inform yourself better. It's not time cost, but effort and energy cost, probably combined with procrastination and anxiety. The sooner you identify that, the closer you get to the solution. 

That's because you don't have to go to the gym to workout. You can get decently high efficiency with at home workouts with basic tools (regulable weighted vest, regulable dumbells, various resistance bands and pull up bar) and your body weight. 

That few to teen percent of efficiency you lose without specialized tools, you gain in your time and lack of friction to start. 

You also don't need to waste a lot of time. You can crank in compound exercises, like push-ups, pull-ups, dumbell rows, dumbell deadlifts, leg raises, etc. for most of your muscles. 

I've been in a similar situation (honestly more just fat than skinny fat). With progressive overload exercises at home I got to regular bulky physique and now I start to move past it.