Projects per year
Abstract
With the growth of participating clients, the centralized parameter server (PS) will seriously limit the scale and efficiency of Federated Learning (FL). A straightforward approach to scale up the FL system is to construct a Parallel FL (PFL) system with multiple PSes. However, it is unclear whether PFL can really achieve a faster convergence rate or not. Even if the answer is yes, it is non-trivial to design a highly efficient parameter average algorithm for a PFL system. In this paper, we propose a completely parallelizable FL algorithm called P-FedAvg under the PFL architecture. P-FedAvg extends the well-known FedAvg algorithm by allowing multiple PSes to cooperate and train a learning model together. In P-FedAvg, each PS is only responsible for a fraction of total clients, but PSes can mix model parameters in a dedicatedly designed way so that the FL model can well converge. Different from heuristic-based algorithms, P-FedAvg is with theoretical guarantees. To be rigorous, we conduct theoretical analysis on the convergence rate of P-FedAvg, and derive the optimal weights for each PS to mix parameters with its neighbors. We also examine how the overlay topology formed by PSes affects the convergence rate and robustness of a PFL system. Lastly, we perform extensive experiments with real datasets to verify our analysis and demonstrate that P-FedAvg can significantly improve convergence rates than traditional FedAvg and other competitive baselines. We believe that our work can help to lay a theoretical foundation for building more efficient PFL systems.
Original language | English |
---|---|
Title of host publication | INFOCOM 2021 - IEEE Conference on Computer Communications |
Place of Publication | Piscataway, NJ |
Publisher | Institute of Electrical and Electronics Engineers (IEEE) |
Number of pages | 10 |
ISBN (Electronic) | 9780738112817 |
DOIs | |
Publication status | Published - 2021 |
Event | 40th IEEE Conference on Computer Communications, INFOCOM 2021 - Vancouver, Canada Duration: 10 May 2021 → 13 May 2021 |
Publication series
Name | Proceedings - IEEE INFOCOM |
---|---|
ISSN (Print) | 0743-166X |
ISSN (Electronic) | 2641-9874 |
Conference
Conference | 40th IEEE Conference on Computer Communications, INFOCOM 2021 |
---|---|
Country/Territory | Canada |
City | Vancouver |
Period | 10/05/21 → 13/05/21 |
Keywords
- Parallel Federated Learning
- Convergence rate
- Network topology
- Mixing matrix
Fingerprint
Dive into the research topics of 'P-FedAvg: parallelizing federated learning with theoretical guarantees'. Together they form a unique fingerprint.Projects
- 1 Finished
-
Building Intelligence into Online Video Services by Learning User Interests
29/06/18 → 28/06/21
Project: Research