Disentangled Loss for Low-Bit Quantization-Aware Training - CEA - Commissariat à l’énergie atomique et aux énergies alternatives
Communication Dans Un Congrès Année : 2022

Disentangled Loss for Low-Bit Quantization-Aware Training

Résumé

Quantization-Aware Training (QAT) has recently showed a lot of potential for low-bit settings in the context of image classification. Approaches based on QAT are using the Cross Entropy Loss function which is the reference loss function in this domain. We investigate quantization-aware training with disentangled loss functions. We qualify a loss to disentangle as it encourages the network output space to be easily discriminated with linear functions. We introduce a new method, Disentangled Loss Quantization Aware Training, as our tool to empirically demonstrate that the quantization procedure benefits from those loss functions. Results show that the proposed method substantially reduces the loss in top-1 accuracy for low-bit quantization on CIFAR10, CIFAR100 and ImageNet. Our best result brings the top-1 Accuracy of a Resnet-18 from 63.1\% to 64.0\% with binary weights and 2-bit activations when trained on ImageNet.
Fichier principal
Vignette du fichier
93.pdf (1.44 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-03776535 , version 1 (13-09-2022)

Licence

Identifiants

Citer

Thibault Allenet, David Briand, Olivier Bichler, Olivier Sentieys. Disentangled Loss for Low-Bit Quantization-Aware Training. CVPR 2022 - IEEE / CVF Computer Vision and Pattern Recognition Conference, Jun 2022, New Orleans, United States. pp.2788-2792, ⟨10.1109/CVPRW56347.2022.00315⟩. ⟨cea-03776535⟩
123 Consultations
154 Téléchargements

Altmetric

Partager

More