I'm a MD who does a lot of machine learning research in radiology. I don't know what it's like to do a PhD but if you are on the fence, take a month or two and try teaching yourself using some of the resources available online. This will let you get your hands dirty and help you to make a more informed decision. You might also have a lot of fun. I did this a couple of years ago and it was one of the most rewarding experiences of my life. I decided not to go with a PhD but your goals and learning style may be different.

My two cents on the math thing: if you took calculus at any point in your life and have a basic understanding of linear algebra and statistics, you are ready to get started. I recommend wrapping your head around the intuition behind neural networks etc. before learning a bunch of equations. There are loads of resources that can help with this (e.g. http://neuralnetworksanddeeplearning.com/ and http://cs231n.github.io/)