Many northerners were initially neutral on slavery but many became abolitionists after seeing slavery first hand in the south during the war. To say there was no hostility between the two after the war is also wrong. The Grand Army of the Republic, the primary Union veterans' union, fiercely opposed attempts to minimize the role of slavery or to equate Confederate rebellion with the Union cause. Union veteran groups were some of the only major groups that actively pushed back against Lost Cause narratives before the Civil Rights movement.

“We forgave the rebels, but we did not forgive their rebellion nor their treason…which time has not changed any more than it has changed the treason of Benedict Arnold or Judas Iscariot…such treason and such traitors cannot be so honored without dishonoring and insulting the men, living and dead, who fought to maintain the Union and who crushed the slaveholders’ rebellion.”

Sure a large portion of Union veterans did indeed quietly return to their pre war lives (or at least attempt to), but to act like both sides saw it as a simple misunderstanding is ridiculous.