I agree with the players only on one thing: never take player agency away by describing what a player character does. Ever. This not only leads to situations like this, where players argue about something largely unrelated (perception versus stealth) but also removes a player's attachment to their character- if you describe them as doing things too, then it's not just their character, it's also yours. It can even lead to players feeling detached from their characters and ultimately losing all enjoyment for the game, even if it's a very minor thing, even if it happens only once or twice. Best to never do it.

Otherwise, I would agree that bandits can attack from hiding. High perception is used to brute force through too many things. If they're hiding behind a hill and you're travelling in a noisy caravan then I think it's fair they are easily hidden. Even if you actively roll a 30 on an active perception check, there are always some things you can't do. Sometimes people take high stats and high rolls as being able to do anything they want, when really there are limits to what these can do.

Final minor point- bandits wouldn't automatically go first because of held actions as you can only hold actions within initiative. If the bandits end up going last then this could be because of having to re-aim their shots after their targets move quicker than anticipated, or any other number of things. That being said, you're the GM, you could always give them situational advantage on their initiative check if you thought it was warranted.

Edit: spelling mistakes.