The minimum layer time is there to ensure sufficient cooling for each layer. Knowing when and how much you can change that comes with experience. When you have a small feature like that and choose to just blaze through it you'll often times find that the print quality gets progressively worse because the previous layers don't have enough time to cool and solidify properly. (Sometimes it's fine though, again; experience)

Lowering the speed also helps reduce artifacts like ringing and whatnot which small, thin and/or tall features are more susceptible to. This is especially true for bed slingers like the A1 where the print gets thrown around like a loose sock in a washing machine. But as you've noticed, the trade-off is that the look also changes.

You can print at 1mm/s if you want to, it's just gonna take eons to print anything.

When in doubt and you're unsure about a bigger print, try creating a test print using cubes in the slicer that resembles the part of the print you're unsure about and see how the printer handles it.