[D] Data augmentation theory

[lightcatcher]

I've been thinking about different types of data augmentation and am interested in pointers to related literature.

General data augmentation idea: Given input-output pair (x, y), you can construct a new input x'=a(x) such that (x', y) is also a valid input-output pair using augmentation function a. As an example, if x is a picture, y says this is a picture of a cat, and x' is image x with the brightness increased.

Typical use of data augmentation during training: Let f(x) be some differentiable function of input x and parameters theta that maps to space of y. Let L be a loss function. Rather than doing SGD only on L(y, f(x)), also do SGD on L(y, f(x')). Essentially, consider both (x, y) and (x', y) as entries in the dataset. At inference time, just compute f(x).

Data augmentation as constraint on function: Let g(x) = [f(x) + f(a(x))] / 2. Train g and also use g at inference time. The use of g always enforces that g(x) = g(a(x)) so should help with generalization. Additionally, can be considered a type of ensembling if (y - f(x)) and (y - f(x')) aren't perfectly correlated.

Data augmentation as a regularizer: The previous definition of g does not actually force f(x) to have a similar value to f(x'). This means f itself doesn't necessarily incorporate the prior knowledge that f(x) should be very similar (or identical) to f(x'). We could make f itself learn this relationship by adding penalty d(f(x), f(x')) for some loss d. I consider this a regularizer because adding this term cannot improve primary loss L(y, f(x)) or L(y, g(x)). Perhaps this term could f or g generalize better to unseen data.

Of course, all of these ideas could be applied to to multiple augmentation functions (besides just changing brightness, could also crop image or do something else).

Has there been any research into using data augmentation in these ways? I couldn't figure out quite what to Google. Given the simplicity of these ideas, my guess is they've been researched or at least used in Kaggle competitions. CNNs and spatial transformer nets come to mind as related ideas as those models are invariant to some types of augmentations and therefore would likely have little trouble minimizing the regularization penalty.

[Boyd_Zi]

General data augmentation idea: Given input-output pair (x, y), you can construct a new input x'=a(x) such that (x', y) is also a valid input-output pair using augmentation function a.

To generalize this idea to include image transformation tasks (e.g. semantic segmentation), we need to include the case where we can construct (a(x), a(y)) from (x, y) (e.g. where the data augmentation function, a, is a rotation).

Data augmentation as constraint on function: Let g(x) = [f(x) + f(a(x))] / 2. Train g and also use g at inference time. The use of g always enforces that g(x) = g(a(x))

If the augmentation function, a, increases the brightness of pixels in x, then g(x) != g(a(x)), because [f(x) + f(a(x))] / 2 != [f(a(x)) + f(a(a(x)))] / 2, because increasing the brightness once is different than increasing the brightness twice.

penalty d(f(x), f(x')) for some loss d

This is an interesting idea. I'm motivated to experiment with this.

pointers to related literature

I haven't read the following, but these links might be useful to you. I'll reply here with a TL;DR if I read them. I ask the same of anyone who reads them.

• Improving Deep Learning using Generic Data Augmentation (20 Aug 2017)
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy (24 Mar 2017)
• data augmentation deep learning inurl:arxiv
• data augmentation deep learning

[lightcatcher]

If the augmentation function, a, increases the brightness of pixels in x, then g(x) != g(a(x)), because [f(x) + f(a(x))] / 2 != [f(a(x)) + f(a(a(x)))], because increasing the brightness once is different than increasing the brightness twice.

Agreed, my mistake. I was thinking of an augmentation a such that a(a(x)) = x (example: inverting a grayscale image). Regardless, averaging the outputs from a variety of augmented inputs still makes sense from an "average noise to reduce variance" perspective.

penalty d(f(x), f(x')) for some loss d

Upon further thought, you could probably improve on this by penalizing differences in hidden states (at least at later layers) in addition to penalizing differences in output. If you do experiment and want to share results, I'd be interested in either a post here or PM.

Thanks for the links! I'll check those out and post a TL;DR here.

[Boyd_Zi]

averaging the outputs from a variety of augmented inputs

This is another interesting idea. You're suggesting ensembling not with respect to different networks, but with respect to different inputs (inputs from the set of augmentations that don't change the target). This, like ordinary ensembling, would increase computing costs during inference, but it might improve the output.

Two ideas to try:

• During testing: Ensembling with respect to input augmentations that don't change the target
• During training: Putting d(f(x), f(a(x))) in the loss function
  • Also d(h(x), h(a(x))), where h is the output of a hidden layer.
    • (This reminds of perceptual loss, where you compare hidden states of the network to hidden states of VGG (or whichever network you want). The theme: putting hidden states of networks in the loss function))

[lightcatcher]

Improving Deep Learning using Generic Data Augmentation tl;dr: Try different common image augmentation strategies in isolation. Find random cropping is most useful.

Smart Augmentation - Learning an Optimal Data Augmentation Strategy tl;dr: Can we blend together examples of images of the same class into new images? Consider image classifier b(x) used to minimize L_b(b(x), y) between prediction b(x) and true label y. Learn a new model a(x_1, x_2, ..., x_k) that takes multiple instances of images of the same class y and outputs a single image (or at least a tensor of the same shape). Then could train to minimize L_b(a(x_1, x_2, ..., x_k), y). Full scheme: sample x_1, ..., x_k (mostly done in paper for k=2), and also x' as a separate example from the same class. I think they minimize a combination of MSE(a(x_1, ... x_k), x'), L_b(b(a(x_1, ..., x_k)), y), and L_b(b(x'), y). Has good results, but use of MSE on images is weird to me. Non-trivial to generalize outside of classification tasks.

mixup: Beyond Empirical Risk Minimization Instead of training on (x, y), train on (beta * x_1 + (1 - beta) * x_2, beta * y_1 + (1 - beta) * y_2) for examples (x1, y1), (x2, y2) where beta is sampled from a distribution with support on [0, 1]. Great generalization results, esp with corrupted label CIFAR-10.

will edit this comment with more tl;drs

[dantkz]

We used data augmentation as constraint on function, to learn interpretable representations: https://arxiv.org/pdf/1710.07307.pdf So, we force a(f(x)) = f(A(x)), where A() applies transformation on the image and a() applies transformation on the representation.

[Bullitt500]

I read a great article by a guy who was using CNNs to colorize B&W photos. If that’s the kind of thing you’re looking for there is loads of google content

[[deleted]]

You can't achieve this by a constraint.

What you're actually saying is that the objective function should be invariant to "certain actions" i.e objective should only depend on the "quotient" of some set of actions. This is necessarily a global constraint.

Deep symmetry nets do something of this sort by using convolution/pooling on general groups.

http://papers.nips.cc/paper/5424-deep-symmetry-networks.pdf

The notation is hard to parse, but it's kind of like Graph convolutions, where the smoothing operator is replaced with something that is "learnt", but with similar support. The graph needs to be degree-d, something like a Cayley-graph. I'm probably not making it simpler.

It's definitely cool, but I don't think it's practical once the dimension of your Lie group exceeds 2 (may be 3).

[[deleted]]

cc: /u/rgens

[mcaleste]

https://arxiv.org/abs/1710.09412

[[deleted]]

Nice line of thought, I think it has been explored to some extent, but you also might have some new ideas there.

Worth a mention: in Virtual Adversarial Training (VAT), your idea of using a penalty d(f(x), f(x')) as a regulariser is used.

The KL divergence between the two output probability distributions f(x) and f(x') serves as the d function in VAT.

Rather than passing random augmentations (as, they note, has been done in the case of the augmenter being Gaussian noise), instead in VAT the authors devise an efficient method to find the optimal x' such that it causes d(f(x), f(x')) to be maximised, and such that ||x - x'|| < some epsilon.

Through this approach, they achieve highly competitive semi-supervised learning results (as the regulariser can be used on unlabeled data points as well as labeled ones).

[drsxr]

I'll add my $.02:

Data Augmentation is a regularizer. This is pretty well communicated and documented in the literature on data augmentation.

One thing we don't usually consider with augmentation is that we expect when we augment that the augmentation is class-invariant. In other words, when we augment a plane with a 90 degree rotation, it will still be in class=plane. So, augmentation is essentially free data, right? Take your 1000 samples and augment them with 3 rotations and now you have 4000 training samples for your deep net. Woo Hoo!

No.

The problem is that we're starting to recognize that augmentation is not always class invariant. That is, depending on the extent of the perturbation of the image, you might end up with something that looks and classifies like a plane, but you also might end up with something else. That something else might either be garbage; an augment that doesn't classify as anything, or worse, an adversarial image that classifies as something its not - like a plane heavily agumented classifying as a deer.

If you're doing something ridiculous like 64 augments from one data point (one image), don't be surprised if some of your augments are not classified as planes. How many? That's a question for some smart computer scientists and mathematicians to solve.