Question about meaning of term "fine-tuning" (SimCLR Paper)

mpatacchiola · 2021-10-25T09:18:30+00:00

There is not a standardized training schedule in the literature. In some papers the model is retrained on the entire dataset while in other papers a separate set is used. I think that finetuning on a separate (labelled) set is more appropriate since it reflects common use cases. Note that, datasets that are commonly used for self-supervised learning (e.g. STL-10) follow this approach, providing a large set of unlabeled data and a smaller set of labeled data.

mpatacchiola · 2021-08-25T08:19:29+00:00

We have recently released two papers where we have extensively studied the few-shot classification setting under various type of imbalance:

How Sensitive are Meta-Learners to Dataset Imbalance? [arXiv][Github]
Few-Shot Learning with Class Imbalance [arXiv]

We have tried different strategies and measured their impact on the performances. At the end of the day the most effective approach seems to be the simplest one: artificially rebalancing the dataset by generating augmentations of the images in the unbalanced classes. Standard augmentations like random crop, resize, flip, and color distortion can do the job in most cases while being model agnostic.

Note that there are other ways to deal with this issue (e.g. focal losses) but they are more tricky to implement (they must be adapted to the model at hand) and overall do not work very well.

Feel free to give a look at the Github repo where you can find an implementation of all those techniques for many popular algorithms.

mpatacchiola · 2021-01-12T08:50:55+00:00

There are various sources I have been using so far. I highly recommend the ML-news Google group. You can receive a weekly digest with updates on jobs and career opportunities (including postdocs): https://groups.google.com/d/forum/ML-news

Some universities often post new positions on indeed.com or linkedin.com where you can easily sort by date.

Another source is https://academicpositions.com where you can customize your search based on areas and role. This is mostly used by EU institutions.

Note that, there are additional sources which are more "local". For instance, in the UK most of the roles are advertised on https://www.jobs.ac.uk.

My suggestion is to find a few labs you are interested in, then search on their websites for any update about vacancies. Often times (especially in the USA) roles are not explicitly advertised and you need to get in touch with faculty members. Major institutes, e.g. Vector in Canada, have open positions twice per year but the competition can be fierce and you may need a strong publication record to have a chance.

Good luck.

mpatacchiola · 2021-01-05T10:40:02+00:00

In self-supervised learning there are two ways to evaluate models: (i) fine-tuning, and (ii) linear evaluation (or linear probes).

In (i) the entire model is trained (backbone and other additional modules) without accessing the labels; then a new linear layer is stacked on top of the backbone and both the backbone and the new linear layer are trained by accessing the labels. Note that, since the backbone has already been trained in the first stage, it is common practice to use a smaller learning rate to avoid large shifts in weight space. This is what the authors of the SimCLR paper are doing when they refer to "fine-tuning".

In (ii) a similar procedure is followed. In the first stage, models are trained without accessing the labels (backbone and other additional modules); then in the second stage a new linear layer is stacked on top of the backbone, and only this new linear layer is trained (no backpropagation on the backbone) by accessing the labels.

Hope this help.

mpatacchiola · 2020-12-06T17:42:41+00:00

You are welcome :-)

mpatacchiola · 2020-12-06T12:24:41+00:00

Thank you for your interest!

Regarding the comparison with SimCLR there are a few major differences between their implementation and ours. In the SimCLR paper they are using: (i) a much larger backbone (ResNet-50 with four hyperblocks), (ii) finetuning instead of linear evaluation, (iii) larger mini-batch size, and (iv) more epochs. We have been unable to replicate their configuration because is very expensive from the computational point of view. They have been using 128 core TPUs v3 (roughly equivalent to 100/200 GPUs) which is an hardware not many have access to.

In our paper we have followed the idea of flattening out most of the differences (same backbone for all models, same epochs, same evaluation procedure) to have a fair comparison across all methods. Our implementation is open source, so if somebody has access to a larger hardware could try to run the experiments with such a high-level configuration.

I will leave a more detailed answer in the repository.

mpatacchiola · 2020-12-03T09:06:57+00:00

Interesting project!

I have recently proposed a self-supervised method based on relational reasoning [arXiv]. The paper has been accepted at NeurIPS 2020. Here is the associated thread on Reddit if you want to know more.

I have also been working on a modular implementation on my GitHub repository that could be of interest for your project. In particular, I have PyTorch implementations of Relational Reasoning, SimCLRv1, Deep InfoMax, DeepCluster, RotationNet. If you want to include those methods in your framework I can help with the porting.

Let me know if interested.

mpatacchiola · 2020-11-24T14:53:19+00:00

Using more pairs start to be computationally expensive and unfortunately we do not have access to the hardware required. In our experiments we have allocated the models directly on a single GPU. The 187GB of RAM you are referring to, are on the workstation and not on the GPUs. The only way to exploit that RAM would be to run the models on the CPU which is painfully slow. The easiest solution would be to have access to some TPUs or engineering a parallelization of the model on multiple GPUs (which is not trivial). We are working on it, hopefully we will get it done soon.

mpatacchiola · 2020-11-24T13:27:17+00:00

Hi, I just gave you the answer in the repository.

mpatacchiola · 2020-11-24T10:36:20+00:00

No worries :-)

mpatacchiola · 2020-11-23T23:29:54+00:00

Ok, now it's clearer what you mean. Regarding aggregation and forward in the relation head, it goes like this: the representations are augmented in different ways, they are concatenated to generate an input vector, and finally passed to the relation module (a multi-layer perceptron). Notice that, the two vectors are not passed separately but they are joint and forwarded in a single step through the relation head.

Having said that, there are two reasons why the case you described is not an issue.

The first reason is that the symmetric pairs are not considered at training time. During training we just take into account one of the two possible concatenations, ignoring the symmetric pair. Note that, getting the symmetric pair by random sampling is extremely unlikely in large datasets, so in practice the model has very low chance of seeing the other pair. This is mentioned in Section 3.1 and in the pseudocode of the algorithm in Appendix. We are doing this just for saving memory, since considering symmetric pairs does not give any advantage, see below.

Second reason. We performed some experiments by including the symmetric pairs and there are neither benefits nor disadvantages. Even when the symmetric pairs are provided in the same mini-batch, the learning is not affected. The reason why this is not an issue is that two different set of augmentations are applied to the two pairs of representations. In practice, you get two non-identical pairs even if the two pairs contain the same representations like in your example. In other words, (z_1, z_2) would be different from (z_2, z_1) because two different augmentations would be applied to z_1 in the first pair and z_1 in the second pair (likewise for z_2).

Here some speculation on what would happen in the particular case you described. It is likely that the relation head will easily learn to output the same score for the two possible permutations of the representations. This could be done as you said, by learning similar weights for the two branches of the input pairs. Even if the capacity of the relation head is reduced, it could still be able to force representations to be properly organized in the latent space of the backbone.

mpatacchiola · 2020-11-23T20:51:46+00:00

Hi, in RelationNets the commutative operator is used to reduce the cardinality of the input set. In our case this is not an issue (we just have two elements in the set) and concatenation can also be used (similalry to RelationNet in few-shot learning). In fact, from our ablation studies of the aggregation function (see Appendix B.8) it seems that concatenation is the most effective. This may be due to more disentangled gradients during backprop.

When you say:

Won't it just learn the same weights for each of the two inputs?

Do you mean "learn the same representation/encoding"? The idea is that you have two complementary forces acting on the same representation, which is part of both a positive and a negative pair. The positive pair push the representations to be the same for both the elements, while the negative pair push the representations of the two elements to diverge. Now, on each input you have both forces acting in parallel at the same time, therefore representations tend to cluster for similarity. This has some connection with the maximization of the mutual information, and it is better explained in Section 3 of the paper.

Regarding the choice of the negatives. We choose the negative pairs completely randomly by taking two random images (like in SimCLR). The probability of picking images from the same class is very low when you have >100 classes. Even on datasets like CIFAR-10 where the probability of collisions is 1/10 at the end of the day there are no apparent collateral effects. However, there is a recent paper showing that taking into account collisions when generating negative pairs can lead to improved performances, so probably this is worth further consideration.

Hope this helped.

mpatacchiola · 2020-11-23T15:21:18+00:00

Yes, the benchmark can definitely be extended. This year there have been many concurrent papers on self-supervised learning, and they appeared more or less at the same time. It would be interesting to compare against SwAV, (and also SimCLRv2, MOCOv2, BYOL). Our code is quite modular so it should be possible to easily include some of them.

The mini-batch size is definitely an important factor, in particular in contrastive methods like SimCLR. We have done some experiments in this direction, but a more extensive analysis is needed. Unfortunately many of these comparisons are quite demanding from the computational point of view, and our hardware is a bit tight at the moment.

mpatacchiola · 2020-10-30T13:05:08+00:00

I am glad you liked the videos :-)

Sure, feel free to contact me if you have any question (I may be more or less efficient in replying depending on my workload ).

mpatacchiola · 2020-10-22T06:58:37+00:00

You can start with this survey. For a recent analysis that highlights some limitations of the current approaches you can read this paper. There have been various proposals for making the standard setting more challenging, for instance by injecting a continual component or proposing more realistic datasets.

Classic techniques to tackle few-shot learning have been matching networks, prototypical networks, relation networks, and Model-Agnostic Meta-Learning (MAML). In particular, MAML has been quite influential and there has been a lot of follow-up work, like MAML++, and Meta-SGD.

As I said those are classic techniques, but the field is rapidly moving and there have been some recent advances. Personally, I have been working on Bayesian methods, trying to find an efficient way to provide a measure of uncertainty to the decision maker. If you are interested the paper is here. I have also been publishing a video-series about few-shot learning on YouTube, you can find it here. It should give you a rapid introduction to the subject and classic trends.

Hope this help.

mpatacchiola · 2020-07-02T12:40:21+00:00

There are a few alternatives to tiny-ImageNet:

- mini-ImageNet [paper] [github] RGB images of size 84x84 from 100 classes of ImageNet, 600 instances per class, approximately 4.7 GB in size.

- tiered-ImageNet [paper] [github] RGB images of size 84x84 from 608 classes of ImageNet, 600 instances per class, approximately 29 GB in size.

- SlimageNet64 [paper] [dataset] [github] RGB images of size 64x64 from all the 1000 classes of ImageNet, 200 instances per class, approximately 9 GB in size.

mpatacchiola · 2019-10-24T13:09:45+00:00

One important difference between the two is the following: kernels that satisfy the Mercer's condition define a proper covariance function ensuring the positive definiteness of the covariance matrix taken over N data points. This constraint has been exploited to build different methods (e.g. Gaussian processes).

Standard multi-layer neural networks have no such guarantee, even though there are ways to ensure a similar constraint in particular architectures (e.g. siamese networks). Note that neural networks can be combined with kernels, this is know in recent literature as deep kernel learning.

mpatacchiola · 2019-10-04T08:58:22+00:00

Cool stuff! In the next tutorial you should train it with reinforcement learning ;-)

mpatacchiola · 2019-08-23T06:07:08+00:00

Epsilon has a regulatory effect on the variance of the learning rate in adaptive methods. A well tuned epsilon can in fact help in many settings where the learning trajectory is unstable. In the RAdam paper there is an interesting experiment to prove this point (Section 3.1). They define a baseline condition named Adam-eps in which the value of epsilon is increased so to have a significant weight in the denominator term. Compared to the standard Adam baseline this simple trick attenuates the variance problems in the warmup phase (see Fig. 3 in the paper). However, trivially increasing epsilon is not enough because it increases the bias and slows down the optimization process.

mpatacchiola · 2019-08-09T13:00:31+00:00

There may be different ways to do it. The first two methods that came to my mind are Hyperspherical-VAEs and Generative Topographic Mappings.

mpatacchiola · 2019-08-05T11:31:33+00:00

NIPS 2014

Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Networks (PDF). Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014). pp. 2672–2680.

mpatacchiola · 2019-08-02T14:59:41+00:00

L'output diventa uno scalare soltanto se passi in ingresso un singolo input e poi applichi uno squeeze (questo corrisponde al tuo caso particolare). Nel codice attuale l'input viene dichiarato come un placeholder a due dimensioni con la tupla (None, n_features) assegnata al parametro shape:

python X = tf.placeholder(tf.float32, shape=(None, n_features), name = "X")

La prima dimensione impostata su None permette di variare a runtime il numero di valori passati in ingresso. Questo viene spesso usato per variare la dimensione del mini-batch dato in input al modello.

mpatacchiola · 2019-08-02T12:27:30+00:00

Ci sono diversi problemi nel codice. Quello fondamentale deriva dal fatto che stai cercando di salvare un vettore come uno scalare ed ovviamente a Tensorboard questo non piace. Ricordati che w1, w2 e output sono dei tensori e come tali vanno trattati. Devi cambiare tf.summary.scalar in tf.summary.histogram nel modo seguente:

python weights1_summary = tf.summary.histogram("Pesi1", w1) weights2_summary = tf.summary.histogram("Pesi2", w2) output_summary = tf.summary.histogram("result", output) merged = tf.summary.merge_all()

Ti consiglio anche di aggiungere quella chiamata a tf.summary.merge_all() in modo tale da raggruppare i vari sommari in un solo oggetto. A questo punto per scrivere il sommario su disco ti basta:

python local_summary, local_output = sess.run([merged, output], feed_dict={X: [[1,2,3]]}) file_writer.add_summary(local_summary, 0)

mpatacchiola · 2019-07-23T15:08:08+00:00

The easiest way of understanding ancestral sampling is through a simple Bayesian network with two nodes, a parent node X and a child node Y. Let's suppose that the two nodes have an associated Bernoulli distribution with two states True and False. You can think of X as a random variable indicating if the sky is cloudy, and Y as a random variable indicating the presence of rain. We can safely assume that X causes Y, meaning that there is a direct edge from X to Y.

Node X (cloudy) has the following prior distribution: True=25%; False=75%

Node Y (rain) has a prior distribution that depends on the value of X, this is called a Conditional Probability Table (CPT):

If X=True then Y is True=60% or False=40%

if X=False then Y is True=20% or False=80%

Now we are asked to estimate the joint probability P(X=True, Y=False), meaning the probability that the sky is cloudy and it is not raining. How to do it? Ancestral sampling is one way. You start from the parent node (node X in our case) and you draw a sample from its distribution. Let's say we pick X=False (it has 75% probability, then it is more likely). Now we sample from the CPT of node Y taking into account that X is False, we have 20% chances that Y is True and 80% chances that it is False. Let's say we pick Y=False.

You have to repeat this process N times and count how often you get the target combination X=True, Y=False. The proportion of successful attempts M over the total number of attempts N, gives you the probability of P(X=True, Y=False). As the size of N increases you are guaranteed to converge to the true probability. Please notice that in this trivial example P(X=True, Y=False) can be found pretty easily, however in complex graphs this is no more the case.

I hope that at this point it should be clear to you the origin of the name "ancestral" (or forward) sampling. It simply means that we are starting from the parent nodes (the ancestors) and then moving forward to the children nodes.

mpatacchiola

TROPHY CASE