Slashing communication traffic in federated learning by transmitting clustered model updates

Laizhong Cui, Xiaoxin Su, Yipeng Zhou*, Yi Pan

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. However, a flexible unbiased compression algorithm applicable for both uplink and downlink compression in FL is still absent from existing works. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS. In MUCSC, it is only necessary to transmit cluster centroids and the cluster ID of each model update. Moreover, we prove that: 1) The compressed model updates are unbiased estimation of their original values so that the convergence rate by transmitting compressed model updates is unchanged; 2) MUCSC can guarantee that the influence of the compression error on the model accuracy is minimized. Then, we further propose the boosted MUCSC (B-MUCSC) algorithm, a biased compression algorithm that can achieve an extremely high compression rate by grouping insignificant model updates into a super cluster. B-MUCSC is suitable for scenarios with very scarce network resource. Ultimately, we conduct extensive experiments with the CIFAR-10 and FEMNIST datasets to demonstrate that our algorithms can not only substantially reduce the volume of communication traffic in FL, but also improve the training efficiency in practical networks.

Original languageEnglish
Pages (from-to)2572-2589
Number of pages18
JournalIEEE Journal on Selected Areas in Communications
Volume39
Issue number8
Early online date7 Jun 2021
DOIs
Publication statusPublished - Aug 2021

Keywords

  • Federated learning
  • convergence rate
  • model update compression
  • clustering

Fingerprint

Dive into the research topics of 'Slashing communication traffic in federated learning by transmitting clustered model updates'. Together they form a unique fingerprint.

Cite this