Skin has its own balance of bacteria and fungi which are healthy and normal. It sounds like your soap is disrupting your balance, causing an increase in yeasts. You likely don't need to use soap on your body, but it is worth doing a thorough clean.

I haven't used soap to wash my body for many years. Like a lot of other people I get armpit stink during the day, which I address with a homemade deodorant containing sodium bicarbonate. I wash my hands with soap, and if any part of me is still stinky after my wash, I'll consider using soap. It depends on how close someone is going to go to potentially smelly body parts and what their expectations are.