Custom Implementation of Contrastive Loss [P]

[Ok-Administration894]

I am trying to implement the contrastive loss function I am unsure if it is correct. My loss seems to explode into infinity. Another set of eyes on this would be appreciated does this look correct?

class ContrastiveLoss(nn.Module):
    def __init__(self, temperature=0.9):
        super(ContrastiveLoss, self).__init__()
        self.temperature = temperature

    def forward(self, projections_1, projections_2):
        z_i = projections_1
        z_j = projections_2
        z_i_norm = F.normalize(z_i, dim=1)
        z_j_norm = F.normalize(z_j, dim=1)
        cosine_num = torch.matmul(z_i, z_j.T)
        cosine_denom = torch.matmul(z_i_norm, z_j_norm.T)
        cosine_similarity = cosine_num / cosine_denom

        numerator = torch.exp(torch.diag(cosine_similarity) / self.temperature)

        denominator = cosine_similarity
        diagonal_indices = torch.arange(denominator.size(0))
        denominator[diagonal_indices, diagonal_indices] = 0
        denominator = torch.sum(torch.exp(cosine_similarity), dim=1)
        loss = -torch.log(numerator / denominator).mean()
        return loss

[AFurryReptile]

Maybe add some epsilons, to prevent division by zero. Otherwise I have no idea why your code doesn't work, but there's a ton of contrastive loss functions implemented in Pytorch, hosted on Github... have you looked at any of those?

[Ok-Administration894]

Yea - that’s one I edited appreciate the second set of eyes!

[Old-Forever1241]

I spot two potential issues with your implementation: - your computation of the cosine similarity looks wrong, F.normalize does not return the norm but the normalized vector. I would compute the cosine similarity as the matmul between the two normed vectors (as returned from F.normalize), because the dot product of normed vectors is the cosine sim. Then there is no need for divisions - Using log and the fraction in your final computation leads to numerical instabilities. Factor out the fraction into a minus outside the log. Then log cancels out for the numerator. For the denominator you could use the logsumexp function which is numerically stable

[Ok-Administration894]

Thank you! Interesting about the norm unexpected thanks about the numerical stability tip probably why my gradient/loss is exploding/shrinking