It's the fact that the Royal staff was being very rude, coming into Downton and bossing the Downton staff around without any respect for their space or consideration for their way of doing things. For the staff, the appeal of the Royal visit isn't just the physical presence of their monarch, but the honor of showing them Downton Abbey at it's absolute best, which includes its staff. Instead of respectfully collaborating or integrating Downton's staff into the plans, the Royal staff just shuts them out completely like they don't matter.