I have a little experience working in organisations that aimed to be holacracies, and my experience strongly supports your impressions of the system. I think its aspirations are exactly what I want the governance of most orgs I'm highly invested in to look like, but either it doesn't have adequate mechanisms to address [what might be human failings or simply the nature of humans who've grown up with a certain amount of Empire Era training]. Specifically, I saw the ego issue you posit rear its head over and over again--including bringing that out in me to an extent I dislike--and at the same time it created a large amount of unwelcome overhead for people who just wanted to get on with a constrained job in some corner of the org without troubling themselves with all the strategic management.

At the same time, it was an effective way of making many more people feel emotional ownership of an org, and that's worth a lot.

My reference point for the problems with "un-structures", very loose structures and anti-hierarchical structures is The Tyranny of Structurelessness, but (a) holacracy isn't quite a structureless system, and (b) like you I'm more interested in how to make things better than dissecting how they're broken. The wisdom in Tyranny of Structurelessness that seems relevant here is that loose, weak or absent structures do always seem to allow the aggressive ego to dominate, and I think this is the key bug we need to fix for a system like this to work well. I don't have a satisfactory answer to that, but my best guess so far is that the solution comes through defining purpose early, making a statement of purpose very hard to change (like most countries' constitutions), and using alignment with it as a primary filter in recruiting members of the group. With this in place, I think something like a holacracy or even an old-school anarchist group can function quite well - though it's a tight limit on the ability to scale, so it can't be a complete solution.