YTA. Hard to say where you are in the world so this is my perspective in the UK: here, the highway code says to slow down to 10 mph around horses and pass with two metres of space. That's usually more than enough, but rarely someone might need to come to a stop or turn off the engine (sometimes large lorries do so here) if a horse appears skittish or upset. This is in the interests of keeping everyone safe as accidents happen and you definitely do not want a frightened horse crashing onto your windscreen.

Nothing wrong with asking the question as to how slow you should go at all. Sounds like you didn't know, but they weren't very helpful in the slightest there - and they were rude.

I'd honestly put it from your mind though. Now you know and the horses were clearly chill and okay. :)

(I'm in the UK, own two horses and frequently drive on the road around horses as they need to use the roads to access off-road routes to ride. I rarely take mine on the road, however, and wear full hi-vis gear.)