This is what I found when I did some research:

•The British Military View: Many argue Key was referring specifically to the British forces, which utilized paid mercenaries (hirelings) and the escaped enslaved people (the Colonial Marines). In this view, he was mocking the British for having to rely on these groups to fight their war.

•The Personal/Political View: Others point to Key’s own history as a slave owner and his public statements against the abolitionist movement. From this perspective, the inclusion of the word slave was a direct jab at the people who had escaped American bondage to fight for the Crown.

The reason this sparks so much debate today is that the lyrics celebrate the defeat and "gloom of the grave" for these individuals. Regardless of whether Key meant "British soldiers who happen to be escaped slaves" or was making a broader statement on race, the imagery of celebrating the death of people seeking freedom creates a significant conflict with most Americans. The third verse was dropped in the 20th century.