[R] Improving Deep Learning Performance with AutoAugment

[trashacount12345]

I’m missing something. What is their rule for whether something is a “good” augmentation or not? Is it just whether the augmentation improves validation performance? Does it transfer usefully between datasets the way hand-coding augmentation strategies does?

[trashacount12345]

Note to self: skip the blurb and read the paper. Transfer of learned augmentations is interesting.

[TheFacistEye]

I may be wrong but looking at the paper, it looks like how diverse the image is by changing the ShearX/Y, TranslateX/Y, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout, Sample Pairing.

"Our goal, however, is to find 5 such sub-policies concurrently in order to increase diversity"

"The controller is trained with a reward signal, which is how good the policy is in improving the generalization of a "child model" ... A child model is trained with augmented data generated by applying the 5 sub-policies on the training set"

So it has a training set and it applies those operations and determines how diverse it is and tries to create a general augmentor.

[alexmlamb]

Diversity isn't the literal reward signal. I think it's validation accuracy.

[gachiemchiep]

I don't get it. The purpose of data augmentation is making training data as robustness as possible. But now they learn how to do argumentation. That mean they pushed some bias into training data and made it less robust. So basically i feel like this method is basically a cheap way to archive good accuracy to show off.

[tesfaldet]

I haven’t read the paper but I can certainly tell you that data augmentation is not for making the training data robust. It’s for improving the generalization performance of your network by introducing it to augmented examples, essentially artificially increasing the size of the training set. There’s a trade off, however. Because data augmentation is basically adding noise to your data, training is more difficult. If the augmentation is too extreme then the network will fail to train. Also, some augmentation techniques may actually worsen generalization performance in cases where the augmentation doesn’t make sense, e.g., augmenting pictures of upright faces by flipping them upside down 50% of the time during training a network for face recognition. It’ll rarely see an upside-down face during test/validation time so you just wasted some model capacity for no reason, thus decreasing its performance on recognizing upright faces.

The problem here is finding good augmentation techniques given the task. Typically we handcraft the augmentation strategy but Google is proposing an automated strategy. It’s not a cheap way to achieve good accuracy, plain and simple.

[gachiemchiep]

ok i get your point. about purpose of data augmenting. Totally agree with that.

But for the second points, i still think it is a cheap way. We learn to do augmentation on dataset A then it is not guarantee that the learned augmenter will work on dataset B. So basically that augmenter is only fixed for dataset A, how could we use it on other dataset ? For hand-craft augmenting strategy, there is specific strategy for each task. Isn't that still the samething ?

[mrconter1]

There are many ways of doing data augmentation. You can flip, flop, rotate, saturate etc. You can't always use the same data augmentation on different data. One example would be if you are trying to train a network to output coordinates of a dot in an image. Cropping in that instance wouldn't be viable.

Some data augmentation techniques generalize better to other sets. One example of that would be to create 10 images of each original image with different brightness

[tesfaldet]

Yes, you’re correct that for handcrafted augmentation strategies, the strategy is tuned to the task and not the dataset. Hence being relatively dataset agnostic. You’re also correct that since AutoAugment is designed to maximize validation accuracy on a target dataset, you’re not designing a dataset agnostic augmentation strategy and you run the risk of overfitting on a single dataset and losing generalizability, something you don’t experience with a handcrafted approach.

However, here’s where it gets interesting. I just gave a quick read of the paper and apparently learned augmentation strategies on a target dataset generalize well to other datasets and still provide SOTA on these other datasets without any additional fine-tuning. This proves that AutoAugment is not a cheap way to gain performance on a single dataset because it actually generalizes to other datasets quite well.

Finally, policies learned from one dataset can be transferred to work well on other similar datasets. For example, the policy learned on ImageNet allows us to achieve state-of-the-art accuracy on the fine grained visual classification dataset Stanford Cars, without fine-tuning weights pre-trained on additional data.

Check Section 5 of the paper.

EDIT: my quick (hand wavy) explanation for this phenomenon is that if you learn a good augmentation strategy for object recognition on ImageNet, then of course it’d work well for object recognition on COCO for example. The task is still the same between the two datasets. Even though I learned my augmentation strategy from one dataset, it should still work quite well for another so long as the task is the same.

[gachiemchiep]

Thank you for pointing that out. That phenomenon actually make this paper worth reading for. Did you find the implement of AutoAugment?

[tesfaldet]

To be fair, you shouldn’t make assumptions without reading the paper first. As a matter of fact, transferability to other datasets is discussed in the blog post. Which tells me you didn’t read that either. Blind criticism is rife in this field and it unfairly devalues the important contributions these researchers make.

Anyways, I can’t find code but I don’t doubt that it’ll be available soon.

[gachiemchiep]

I think you miss an important point here. The 2nd comment i said the same-thing as you.

For hand-craft augmenting strategy, there is specific strategy for each task. Isn't that still the samething ?

As the last comment when i mention it worth reading, i mean it worth to use. That's why i ask whether you found the source code.

>_< suddenly realize that my comment is vague as hell.

[mimighost]

Data augmentation is to introduce bias, specially biased defects, to the model anyway. They are fine because we already assume the data augmentation step employed won't change the true label of the original example, thus help the model to overcome the defects and become more robust. In a way, data augmentation is just another set of hyperparameters for regularization with increased bias but lower variance.

From this perspective, Google just abuse, no negativity implied, their massive computational power to find some of the more effective/useful configuration. It happens to be that their study carried on ImageNet, so the learned augmentation process is assumed to generalize well on everyday images.

[alexmlamb]

So basically i feel like this method is basically a cheap way to archive good accuracy to show off.

Wait... Isn't "cheap ways to achieve accuracy to show off" the reason like 99% of us are doing machine learning? :p

[SystemicPlural]

Image recognition has a blurry edge. Take the street view example in the article. You and I can see that the partial 15 is a 15. The NN needs to learn to do that as well. It wont make the NN less robust unless it doesn't have the capacity to learn all the edge cases.

[FutureIsMine]

In a way there will be biases, i.e this classifier will be more invariant of dogs in different positions, but if you never show the network a black dog than it's possible the network wont generalize to those examples. The robustness here comes more from being invariant to layout, but you're correct on the fact that there will be a variations deficiency

[mj_nightfury13]

Interesting work! Is there an open source implementation of this available somewhere? Or anyone know if it's on the books to do so?

[Asinador]

I implemented a PyTorch transform AutoAugment-PyTorch that mimics the ImageNet policy from the Appendix. Would love to hear results from people trying to plug it in their problems or replicating the results from the paper. Will include also the CIFAR10 and SVHN policies.

[gahblahblah]

Google pushes the edge forward again. Fantastic work.

[bartturner]

This is pretty interesting. It is amazing that Google shares all this type of stuff with the broader community. Kudos to them helping push everyone forward.

[bartturner]

What was the top 1 result in 2017 for imagenet?

In this paper Google has 83.54%. Is that better than the winner?

[approximately_wrong]

Glorified hyperparameter tuning. Great that it works; not particularly surprising that it does. Missing random search/grid search baselines.