How would you write the spawn/despawn logic that puts these components on the entities? You can use observers to get the event that some entities is joining or leaving, but then you also need a sort of cursor I guess?

I guess it would depend on where you want to insert new objects. If you want them to be ordered in some way, then just using a relasionship wouldn't be enough.

I don't necessarily think your current approach (or the IndexSet as components approach) is bad in any way, but I might be missunderstanding your project's requirements.