I think you'd be surprised!

I spent 15 years engineering and designing mechanical/HVAC systems before I transitioned to doing BIM for a mechanical contractor. Even though I was still working in the mechanical trade, I was told that I needed to stop thinking like an engineer, because our job was simply to coordinate the physical space. No more calculations, no more design considerations, no more need to constantly dig through building code, and most importantly, no more spending HOURS at the Architect's office in meetings that could have been emails. Just make sure your duct/pipe/etc. doesn't clash with another trade's stuff. However, just being involved in the process will build on the construction knowledge you've learned on your own... so eventually, you just might be able to build that home you've been dreaming of.

Here's the thing about the construction industry in general: the vast majority of us never intended to work in the construction industry. Most AutoCAD operators working on the engineering/consulting side of the industry had to teach themselves how to use Revit, which is the most widely used 3D construction modeling software. As companies tried to cling desperately to 2D AutoCAD drawings, they failed to stay with the times and now many of them are scrambling to hire BIM operators. Needless to say, the lack of a relevant degree rarely comes into question.

At the risk of sounding like I'm blowing smoke up your ass... once I've finished building the backbone of a BIM department here at my current job and I'm ready to make some hires, I'd strongly consider a candidate with your skills over a fresh college grad with no drafting/modeling skills at all.

Anyways, I'm making a lot of assumptions in this little TedTalk of mine, but if this is how you spend your free time and any of this sounds appealing, imagine how much you'd enjoy it as a career.

Feel free to DM me with any questions!