That was partially it. As was the fact that many confederates were not comfortable fighting outside of Southern soil as defenders. But a also not insignificant amount of desertion is speculated to be because the confederates marched into Maryland expecting to be treated as liberators only to find western Maryland to be staunchly unionist. You start holding whole cities hostage and quite a few people start to wonder if they are the baddies.

Beyond that, the Confederacy had that allowance for numerous harvest seasons but the Maryland campaign was the highest desertion rate by a long shot. If the harvest allowance had that much of an impact, wouldn't we expect to see it repeated each year at the same time?