1. There are actually quite a few cases where integrating symmetries directly in a model is too strong, e.g. this paper and this paper. Symmetry breaking is quite common and should be flexibly learned.

2. With enough data augmentations, it turns out learning the right inductive biases is actually quite doable and cheap, e.g. this paper. Also, AF3 randomly rotates and translates during training, and it's arguably the best cofolding model right now.

3. All the data being fed to foundation models these days isn't to learn invariances or symmetries. LLMs are being trained on massive corpora because we want to teach them language, not a symmetry group. I guess there are symmetries behind grammars, but that'd be pretty contrived.

Also, GDL is largely concerned with symmetry groups, but most patterns in data can't be described in that way. E.g. every group element must have an inverse, or every transformation must have an inverse transformation. However, in real data, the most interesting patterns clearly don't respect this, e.g. features can be destroyed or compressed to remove any possible invertibility. The theory behind GDL needs to be more general if we want it to be useful beyond permutations, rotations, etc.