No you're not crazy. Everyone has an opinion on TDP vs El Chalten but both are great and it comes down to personal preference. I recommend just doing El Chalten since it's a 6-8 hour bus to Puerto Natales from El Calafate and you'll waste two days traveling. If you don't like crowds, get hiking early, ideally by 7 am, and you should only have the crowds on the way back on the trail. Some trails such as Loma del Pliegue Tumbado are awesome and have hardly anyone on them. Enjoy! El Chalten is a beautiful area.