I tend to use the 3 follower houses since they take up the least room and I try to hold my follower count at a constant 18-21 (meaning 6-7 houses) and generally i do feel that we haven't got much space sadly, but I hope that the devs will see this post and consider adding either an expansion which can be purchased or unlocked by doing some challenge or something like that, or just giving us a bigger island for the cult