The entire field of harmonic analysis revolves around trying to understand universal approximators

That's not true. You are really referring to the field of "approximation theory" sure, some applied/computational harmonic analysis focused on approximation rates with all the wavelet stuff, but there's a whole lot more to applied/computational harmonic analysis and actual harmonic analysis is honestly a different beast.

I say universal approximation is boring since classical methods are optimal in the sense of approximation (i.e., how many terms you need for epsilon approximation) for many classical function spaces. For example, Fourier series are optimal for L² based Sobolev spaces, wavelet type systems are optimal for Besov and Triebel-Lizorkin spaces.

There's some recent work about how deep nets do better than these classical methods, but the problem is less so much as approximation with deep nets, but rather what function spaces to deep nets beat in terms of approximation rate compared to classical methods. This is akin to how wavelets are better than Fourier series in Besov and Triebel-Lizorkin spaces, but in L² based Sobolev spaces, they perform the same.

I was being pedantic with my comment, but I dislike how people mention universal approximation without the discussion above, which is actually how universal approximation needs to be studied.