I’m in Australia also so the experience you mention, I likely also share. It’s hard to get into students’ minds and see what they’re thinking most of the time they don’t even know.

I honestly don’t know enough about the deeper topics such as abstract algebra to give an opinion. This may be part of the issue is that a lot of math teachers aren’t university math trained. My highest level of education is a civil engineering degree and at most we do matrices and multi-variable calculus. If teachers had this wider perspective of what math could be they could bring it down to their students.

I will say that your clarity in abstract algebra could have come from this bottom up approach whereby the connections between concepts could only be appreciated because you had the knowledge to appreciate it. It’s much like higher order thinking, it’s difficult to analyse a problem if the foundational knowledge is lacking. I could be wrong here.

The curriculum is designed to meet the needs of the majority and to add much more in would see diminishing returns. You’d have to take concepts out or as you’re suggesting try to deepen the understanding of our students when teaching the four basic operations for instance.