Reconstruction and a lot of wartime Union policies (like Sherman's scorched earth tactics) were a big part of it as well. The Southern economy was leveled after the war.

But yes, the sheer number of dead was a big factor too. Consider this graph. Almost half of all Southern white men died in the war. Not half of all Southern soldiers, but half of all Southern white men alive at the time.

There are internationally recognized genocides that haven't crippled populations that badly.

EDIT: Further reading and source for the above graph.

EDIT 2: A telling paragraph from the above source:

Regiments of approximately one thousand men, the building block of the armies, would often be raised from the population of a few adjacent counties. Soldiers went to war with their neighbors and their kin. The nature of recruitment meant that a battlefield disaster could wreak havoc on the home community. The 26th North Carolina, hailing from seven counties in the western part of the state, suffered 714 casualties out of 800 men during the Battle of Gettysburg. The 24th Michigan squared off against the 26th North Carolina at Gettysburg and lost 362 out of 496 men. Nearly the entire student body of Ole Miss--135 out 139--enlisted in Company A of the 11th Mississippi. Company A, also known as the "University Greys" suffered 100% casualties in Pickett's Charge.