Disentangled Loss for Low-Bit Quantization-Aware Training - CEA - Commissariat à l’énergie atomique et aux énergies alternatives
Communication Dans Un Congrès Année : 2022

Disentangled Loss for Low-Bit Quantization-Aware Training

Résumé

Quantization-Aware Training (QAT) has recently showed a lot of potential for low-bit settings in the context of image classification. Approaches based on QAT are using the Cross Entropy Loss function which is the reference loss function in this domain. We investigate quantization-aware training with disentangled loss functions. We qualify a loss to disentangle as it encourages the network output space to be easily discriminated with linear functions. We introduce a new method, Disentangled Loss Quantization Aware Training, as our tool to empirically demonstrate that the quantization procedure benefits from those loss functions. Results show that the proposed method substantially reduces the loss in top-1 accuracy for low-bit quantization on CIFAR10, CIFAR100 and ImageNet. Our best result brings the top-1 Accuracy of a Resnet-18 from 63.1% to 64.0% with binary weights and 2-bit activations when trained on ImageNet.
Fichier principal
Vignette du fichier
93.pdf (1.44 Mo) Télécharger le fichier
Disentangled_Loss_Quant_final.pdf (95.48 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-03689954 , version 1 (12-11-2024)

Identifiants

Citer

Thibault Allenet, David Briand, Olivier Bichler, Olivier Sentieys. Disentangled Loss for Low-Bit Quantization-Aware Training. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun 2022, New Orleans, United States. ⟨10.1109/CVPRW56347.2022.00315⟩. ⟨cea-03689954⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More