Beyond parameters: locally-guided knowledge distillation for decentralized federated learning

Federated Learning (FL) has revolutionized privacy-preserving machine learning by enabling collaborative model training across multiple clients without sharing raw data. However, server-based FL suffers from scalability issues, communication bottlenecks, and the risk of a single point of failure. As an alternative, Decentralized Federated Learning (DFL) eliminates the need for a central server through a peer-to-peer setup. Yet, it faces significant challenges due to non-IID (non-independent and identically distributed) data across clients, leading to unstable convergence and poor generalization. A fundamental dilemma in DFL arises from this non-IID nature: aggregating model parameters to promote global consistency can erode clientspecific knowledge, while conducting local training exacerbates model drift, misaligning clients from shared objectives. To address this, we propose BlendDFL, a novel DFL framework that enhances generalization performance by integrating locally-guided knowledge distillation into local training. Specifically, rather than merely minimizing a local loss function, each client minimizes a synthetic loss, which combines the knowledge distillation loss (preserving useful information from neighboring models) and the local model loss. This approach mitigates the overwriting of peer knowledge during local updates. Besides, considering class imbalance and data heterogeneity due to non-IID data distributions, BlendDFL introduces adaptive per-sample weighting, emphasizing underrepresented classes in both types of loss computation. Importantly, BlendDFL incurs no additional communication overhead and does not require access to public data. Extensive experiments on benchmark datasets demonstrate that BlendDFL consistently outperforms the state-of-the-art DFL baselines, achieving faster convergence and better generalization across diverse non-IID settings. The source code of our work is available at https://github.com/behnazsoltani/BlendDFL.