Similarity contrastive estimation for self-supervised soft contrastive learning - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Similarity contrastive estimation for self-supervised soft contrastive learning

Résumé

Contrastive representation learning has proven to be an effective self-supervised learning method. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as positives that should be contrasted with other instances, called negatives, that are considered as noise. However, several instances in a dataset are drawn from the same distribution and share underlying semantic information. A good data representation should contain relations, or semantic similarity, between the instances. Contrastive learning implicitly learns relations but considering all negatives as noise harms the quality of the learned relations. To circumvent this issue, we propose a novel formulation of contrastive learning using semantic similarity between instances called Similarity Contrastive Estimation (SCE). Our training objective is a soft contrastive learning one. Instead of hard classifying positives and negatives, we estimate from one view of a batch a continuous distribution to push or pull instances based on their semantic similarities. This target similarity distribution is sharpened to eliminate noisy relations. The model predicts for each instance, from another view, the target distribution while contrasting its positive with negatives. Experimental results show that SCE is Top-1 on the ImageNet linear evaluation protocol at 100 pretraining epochs with 72.1% accuracy and is competitive with state-of-the-art algorithms by reaching 75.4% for 200 epochs with multi-crop. We also show that SCE is able to generalize to several tasks.
Fichier principal
Vignette du fichier
0168_before_ieee_NoteIEEE.pdf (372.62 Ko) Télécharger le fichier
0168-supp_NoteIEEE.pdf (4.32 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-04166419 , version 1 (19-07-2023)

Identifiants

Citer

Julien Denize, Jaonary Rabarisoa, Astrid Orcesi, Romain Hérault, Stéphane Canu. Similarity contrastive estimation for self-supervised soft contrastive learning. WACV 2023 - 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, IEEE Computer Society, Jan 2023, Hawaii, United States. pp.2705-2715, ⟨10.1109/WACV56688.2023.00273⟩. ⟨cea-04166419⟩
31 Consultations
14 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More