A comprehensive study of transfer learning under constraints - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

A comprehensive study of transfer learning under constraints

Résumé

Pre-training on an upstream task is widely used in deep learning to boost performance of downstream tasks. Recent studies analyzed pre-training with large datasets and large deep neural network architectures. However, pre-training is very useful in practice when downstream tasks have scarce data and are trained under computational constraints. To assess pre-training performance in this setting, we train different deep architectures with 1M parameters. We create different subsets of ImageNet to study the influence of upstream dataset in detail by varying the total size, but also the ratio between number of classes and samples per class for a constant total size. Then, we use the resulting models in transfer toward six diversified downstream tasks using linear probing and full fine tuning for downstream training. Experimental results confirm previous ones regarding performance saturation in downstream tasks, but we find that saturation occurs faster for compact deep architectures. The use of different ImageNet subsets leads to globally similar performance when enough data is included, regardless of the dataset structure. The comparison of downstream training strategies shows that linear probing can be competitive, particularly for few-shot settings. This is at odds with previous reports, which assert the superiority of full fine tuning. Finally, we observe that the type of deep architecture has a significant effect of results, but that their relative performance varies depending on the downstream training strategy.
Fichier principal
Vignette du fichier
Study of Transfer Learning under Constraints v3.pdf (1.35 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-04463322 , version 1 (16-02-2024)

Identifiants

Citer

Tom Pegeot, Inna Kucher, Adrian Popescu, Bertrand Delezoide. A comprehensive study of transfer learning under constraints. ICCVW 2023 - 2023 IEEE/CVF International Conference on Computer Vision Workshops, Oct 2023, Paris, France. pp.1148-1157, ⟨10.1109/ICCVW60793.2023.00123⟩. ⟨cea-04463322⟩
13 Consultations
16 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More