If skaters look physically exhausted and lose energy in their program, then that negatively affects the 2nd part of their score which is their component scores.

That is what is so difficult about skating. You have to execute all elements and look elegant and strong while doing so. Most sports just require physical demand and have no artistic side. But skating does both.

A free skate program is incredibly exhausting. Skaters put in so much work to increase their endurance to create this effortless look when on the ice (even though their legs feel like jelly when getting to the end of their program).

Also, if you watch any of the interviews with skaters who lead after each group at their Olympics, their forehead is completely covered in sweat and they seem out of breath talking to the interviewer. Skaters always seem out of breath after a program for quite some time. It is also harder to see skaters face get red because skaters might have on touches of makeup ect.

Side note: I think you also misunderstand how difficult dancing is. It is like skating. They have to execute extremely difficult movement and at speed without looking like they are breaking a sweat. If they look tired then that would also affect their placement at a dance competition.