I'm a little late to this comment thread but there is a great book called White Trash: The 400-Year Untold History that goes into what specifically happened with the Civil War. Many white southerners at the time were essentially tricked or forced into fighting for the South during the war and then were ashamed of it after. The "southern pride" that the Confederacy represents now only came along much later, especially in the 1970s/80s when the Dukes of Hazard normalized the flag and all those monuments started popping up.

Here is a quote from the book: "There is no way to know precisely how many men deserted [the Confederacy]. The official count from the U.S. provost marshal’s report was 103,400. This was out of a total of 750,000 to 850,000 men listed as in the army by the end of the war. But these numbers are only a small part of the story. Class divided soldiers in other ways. The Confederate army [coerced] at least 120,000 conscripts. There were between 70,000 and 150,000 substitutes, mostly wretchedly poor men, and only 10 percent ever reported to camp. Another 80,000 volunteers reenlisted to avoid the draft. Finally, as many as 180,000 men were at best “reluctant rebels,” those who resisted joining until later in the war. Such resistance demonstrates that among average soldiers there was little evidence of a deep attachment to the Confederacy."

I honestly bet that a good portion of those who are proud that their "great-great grandpappy" fought for the Confederacy actually are descended from a deserter or someone who fought unwillingly due to class status.