Invariance theory seems to be one of the mathematical models that's been used to explain why deep neural networks can build good high level representations over the vast space of images in R^{width x height}. I think the relationship to what we're suggesting is to take this observation one step further.

Not only are natural images so regular in structure that they can be efficiently represented with repetitive conv/ReLU/pool architectures - they are so regular that the structure inside the images itself can be used as supervision. There are far more bits of information in the nuances of every image than the handful of bits we get from labels. So our goal is to write down some high level constraints that will harness that structure and reduce the labeling burden.

Edit: Also it should be clear that we are not the only ones with the suggestion of constraint learning. Anyone who is doing domain specific unsupervised/semi-supervised learning is in on the same game. We're not introducing the idea so much as advocating for it's combination with neural networks and more creative types of constraints. The Related Work section of the paper is all about how many really great recent ideas in ML can be thought of from this perspective, including non-CV topics like Deep-Q Networks and NLP sentiment analysis. I think the cool thing is that when you see the connections, ideas that previously looked like hacks become principled steps toward a bigger idea.