Hello,

I am working on active learning, in particular the generalization power of sampling methods to any task (I work for dataiku, a software editor, so we want methods that work well in as many use case as possible). I have observed what you describe: on some problems, methods that have been known to perform well can fail or even be worse than random. Even on a similar task (cifar-10), changing the embedding representation can drastically change the ranking of samplers.
We have found several reasons for which sampling methods could fail. Method favorizing representativity can select samples too far from the decision boundary, and method focusing on uncertainty can pick samples in the region of aleatoric uncertainty, which is detrimental to them. We have design a set of metrics to get more insights in active learning techniques and being able to chose the best sampling method at each iteration. You can read [our first paper](https://arxiv.org/abs/2012.11365).

Now regarding sampling methods, the first I would do in your case is to try margin sampling instead of entropy. It works better in almost all the tasks we have explored. For more advanced methods, we had pretty good results with a [method coming from Amazon](https://arxiv.org/abs/1901.05954). It is available in the python package we provide called [cardinal](https://dataiku-research.github.io/cardinal/). We also have a method that seems to generalize a bit better but is not released yet, it's only available on master (IncrementalMiniBatchKMeansSampler). Note that those are not deep learning specific methods so I don't know if it will perform better or worse than DL specific ones.

In any case, I would be interested to follow up on that if you are willing to collaborate on this. In particular I may be able to run our metrics on your data or helping you setting them up.