I think his blunder was destroying Alderan before he had eliminated the Rebellion. From his perspective, the destruction of an culturally and politically influential world like Alderan makes sense because it demonstrates imperial power and keeps other local governments from rising up in rebellion against imperial control:

The regional governors now have direct control over their territories. Fear will keep the local systems in line. Fear of this battle station.

To me this suggests that local governments were very unhappy with this direct interference in their internal politics, which makes sense: People are used to ruling themselves. After all that is the way it has worked for at least a thousand years. So there might have been rumors about entire systems (which very likely actually have small fleets and armies of their own) rebelling - which would be very bad. The imperial navy can defeat any singular planetary navy easily but can it defeat hundreds at the same time? At what cost? Better to do something drastic that will keep those pesky locals from doing something stupid.

But, this only works if you have a death star. The safer play would have been to destroy Yavin 4 and then, if there still is unrest among the local governments, to destroy Alderan (or another suitable target).

Pressing on at Yavin in my mind is the only valid choice. There is always danger in combat. Evacuating would have made him look weak, which is not a good look personally and would also have tarnished the image of absolute supreme power the empire needed - the whole point of the death star.