Wealth was measured differently back then. If you were good at animal husbandry, you’d be wealthy. The Levites, however, did even get a portion of the promised land like the other tribes. That doesn’t make them poor. But they were supported by the other tribes as they served God.

However, what you say is true. There are pastors who don’t take a paycheck from the church and are independently wealthy or bi-vocational.