SBNN: slimming binarized neural network

Qing Wu, Xiaojin Lu, Shan Xue, Chao Wang, Xundong Wu, Jin Fan*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

With the rapid developments of deep neural networks related applications, approaches for accelerating computationally intensive convolutional neural networks, such as network quantization, pruning, knowledge distillation, have attracted ever-increasing attention. Network binarization is an extreme form of network quantization technique, which binarizes the network weights and/or activation values to save computational resources. However, it often introduces noises into the network, and requires larger model size (more parameters) to compensate for the loss of representation capacity. To address the model complexity reduction challenges and further improve the network performance, this paper proposes an approach: slimming binarized neural networks (SBNN), which reduces complexity of binarized networks with acceptable accuracy loss. SBNN prunes the convolutional layers and fully-connected layer in a binarized network. Then it is refined by the proposed SoftSign function, knowledge distillation and full-precision computation to enhance the network accuracy. The proposed SBNN can be also conveniently applied to a pre-trained binarized network. We demonstrate the effectiveness of our approach through several state-of-the-art binarized models. For AlexNet and ResNet-18 on ILSVRC-2012 dataset, SBNN obtains negligible accuracy loss but even a better accuracy than the pre-pruning model while using only 75% of original filters.

Original languageEnglish
Pages (from-to)113-122
Number of pages10
JournalNeurocomputing
Volume401
DOIs
Publication statusPublished - 11 Aug 2020
Externally publishedYes

Keywords

  • Binarized neural network
  • Network pruning
  • Knowledge distillation

Fingerprint Dive into the research topics of 'SBNN: slimming binarized neural network'. Together they form a unique fingerprint.

Cite this