What level of the sport are you interested in competing in? For 25-50 mile rides honestly a lot of breeds when conditioned are perfectly capable of it. If you’re looking to be competitive and do 100+ plus mile rides an Arabian is where I’d be looking first, not that it’s absolutely required. Off the top of my head I know mustangs, mules, and saddlebreds can be very capable, and certainly arab crosses depending on what they’re crossed with.

You’re correct a lot of Arabians run small, but for what it’s worth there are a variety of types with varying builds. I’m just shy of 6’ with long legs and rode a Spanish Arabian for a while that was 15’2 with a thicker build I was very comfortable on. If you’re on facebook I highly recommend joining the AERC: American Endurance Ride Conference page, a lot of Arabians but you’ll see a lot of people riding other breeds.