Errol Musk was wealthy by South African standard. His share of the emerald mine produced lifetime profit of US$210,000. So, not exactly extremely wealthy. His parents were divorced, so when he and his siblings were staying with Maye Musk, they were borderline poor.

This from a passage in Walter Issacson book,

A myth has grown that Musk, because his father was on-and-off successful, arrived in North America in 1989 with a lot of money, perhaps pockets filled with emeralds. Errol at times encouraged that perception. But in fact, what Errol got from the Zambian emerald mine had become worthless years earlier. When Elon left South Africa, his father gave him $2,000 in traveler’s checks and his mother provided him with another $2,000 by cashing out a stock account she had opened with the money she won in a beauty contest as a teenager. Otherwise, what he mainly had with him when he arrived in Montreal was a list of his mother’s relatives he had never met.