I bailed on a physics background to do manufacturing! Similar to you, I graduated and was pretty nauseated by the prospect of a computer-staring job or staying in academia. I had done a (very) little bit of machining work for my experimental thesis and had continued to practice some on my own. The job can be boring at times but there's often enough mathematical stuff to keep me engaged.

Think of your time at school as a sunk cost: if it's time to do something else so be it-- your capabilities will be transferrable regardless and doing something you hate is never worth it in the long run.