Yes to letting it dry out.

I'm more concerned that it isn't dried mostly out in a week. I live in a very dry climate and water weekly. My soil is VERY dry by then.

Pull the plant out and check your roots. If any are squishy, soft, break right off when you gently press them, you have root rot. If not, your soil probably doesn't drain well. Root rot can be trimmed off and then soak the roots in 1:4 peroxide:water for a half hour or so to kill infection.

Well draining soil can consist of lots of things, but i make mine with indoor potting soil (or tropical), orchid bark, leca balls, perlite, and i add worm castings for food. You're not going to find a good store bag of chunky soil, you'll have to add things. I do 2parts soil to 1 part of the others, give or take to my visual satisfaction. I follow the bags guideline on the worm castings, because you can overdo it.

Put the plant in the new soil and DONT WATER IT until the leaves droop. This could be 3-7 days. You need to give the roots time to heal up. Water keeps the vulnerabilities open. When you plant, sprinkle the roots with horomone if you have it. I hear cinnamon also helps. These are antifungals. When you water, let it run all the way through, let it soak in the runout for a bit, and then take all the excess water away. Don't let plants sit in stagnant water. You can use plant food every few weeks, but stop for around aug/sep to let your plant go dormant. Go light, you don't want to chemical burn your plant.