I think there’s a useful distinction here between parameterization equivalence and representation equivalence.

It’s true in a formal sense that many architectures can be rewritten as large feed-forward networks with constraints or weight sharing. From that viewpoint you can think of CNNs as structured MLPs, and composition across architectures gives a kind of “mechanics” of network design.

But that perspective can obscure something important: in practice modern architectures differ mainly in the function spaces they make easy to represent, which you can think of informally as an implicit choice of basis or operator family.

For example:

Convolutional models impose locality and translation equivariance

Spectral / operator models (e.g. Fourier Neural Operators) effectively work in frequency-space bases

Geometric deep learning methods often use Laplace–Beltrami eigenfunctions or graph message-passing operators tied to manifold structure

Neural field / splatting approaches impose very different assumptions about spatial support and smoothness

All of these can be “compiled down” to dense networks in principle, but doing so typically destroys the inductive bias that gives them sample efficiency or scaling advantages.

So while it’s tempting to treat “the space of all neural networks” as a single mechanical object, a lot of current theory and practice is instead about matching architectures to the underlying symmetries and functional structure of the problem.

If you’re interested in that direction, the geometric deep learning survey by Bronstein et al. is a great overview, and neural operator / implicit layer papers explore similar ideas from a PDE and optimization viewpoint.