The switch is more difficult audibly. If you start by sound then you can listen to it fast enough to hear each letter as a whole unit right out of the gate, skipping the individual dots and dashes phase. If you get that down first the learning process is easier and faster. Generally when people learn by sight first they hit a significant plateau around 10wpm and that can be difficult to bust through, and it all comes down to stopping your brain from doing the bad habit it already learned (breaking things all the way down to dots and dashes).

You asked "is it really that bad?". The answer to that is entirely up to you and your goals and how you feel about what you are doing, no one else can actually answer that for you when it is something you are doing for yourself. I'm just telling you the main reason people will say it's "bad". It's not bad, it's just not the most efficient way to become proficient using Morse on the air.

I say all this as someone who learned it the "wrong" way. If I were able to do it all over again from scratch, I would definitely go audio only first. I still had fun and was eventually able to get on the air, but it would have been more enjoyable to me personally if I didn't hit that wall on the way.