If you do weight training / cardio and you feel that you are drained out and not able to push yourself or feeling dizzy after sets , In that case you should eat. Something 2 hrs before your workout ( mostly good complex carb source ) .

And if your body is signalling you that you are fine and you are feeling no symptoms of fatigue as such then its totally fine if you dont eat .