通信高效的自适应联邦剪枝优化方法

Adaptive Federated Pruning Optimization Method with Efficient Communication

摘要

联邦学习中深度神经网络的参数量巨大,每轮训练客户端需上传完整模型更新参数,在带宽受限环境下,通信开销成为系统性能瓶颈,尤其是在带宽受限环境下.因此,在保证模型性能的同时降低通信开销是联邦学习研究的关键问题之一.针对上述挑战,本文提出了一种通信高效的自适应联邦剪枝优化方法(communication-efficient adaptive federated pruning optimization method,CEAFL).核心是阶段式自适应模型剪枝算法,分为初始剪枝和自适应剪枝两个阶段,用梯度重要性进行模型剪枝,实现轻量化传输.此外,设计了集成分类器复用的模型微调算法,提升泛化能力和数据分布感知能力.实验表明,相较于基准方法,该方法在多个数据集上的模型精度提升了超过0.5%,同时通信量减少了约38%,展现了其在实际应用中的潜力.

Abstract

The number of parameters of deep neural networks in federated learning is huge.The client needs to upload the complete model update for each round of training,which makes the communication overhead become the bottleneck of system performance,especially in bandwidth-constrained environments.Therefore,reducing communication while ensuring model performance is one of the key issues in federated learning research.In response to the above challenges,this paper proposes a communication-efficient adaptive federated pruning optimization method(CEAFL).The core is a staged adaptive model pruning algorithm,which is divided into two stages:initial pruning and adaptive pruning.The model is pruned using gradient importance to achieve lightweight transmission.In addition,a model fine-tuning algorithm for integrated classifier reuse is designed to improve generalization ability and data distribution perception ability.Experiments show that compared with the benchmark method,this method improves the model accuracy on multiple data sets by more than 0.5%,while reducing the communication volume by about 38%,demonstrating its potential in practical applications.

关键词

联邦学习 / 模型压缩 / 模型裁剪 / 知识蒸馏

Key words

federated learning / model compression / model pruning / knowledge distillation

[1] McMahan B,Moore E,Ramage D,et al.Communication-efficient learning of deep networks from decentralized data[C]//Artificial Intelligence and Statistics,2017:1273-1282.
[2] Xu J,Glicksberg B S,Su C,et al.Federated learning for healthcare informatics[J].Journal of Healthcare Informatics Research,2021,5(1):1-19.
[3] WANG R J,WANG J B,ZHANG F L,et al.Feature map poisoning attack and dual defense mechanism for federated prototype learning[J].Journal of Software,2024,36(3):1355-1374.
[4] Islam M M,Alawad M.Efficient federated learning through distributed model pruning[C]// IEEE Computer Society Annual Symposium on VLSI,2024:155-160.
[5] CHEN X,QIU H B,LI Y L.Edge-assisted adaptive sparse federated learning optimization algorithm[J].Journal of Electronics & Information Technology,2025,47(3):1-12.
[6] Wang D W,Hsieh C K,Chan K L,et al.Model pruning for wireless federated learning with heterogeneous channels and devices[C]//VTS Asia Pacific Wireless Communications Symposium,2023:1-5.
[7] Huang Y,Chen W,Zhu S,et al.Fed-STP:an improved federated learning approach via stage-wise training and pruning[C]//2nd International Conference on Cloud Computing,Big Data Application and Software Engineering,2023:108-113.
[8] Yang Y.Towards superior pruning performance in federated learning with discriminative data[J].IEICE Transactions on Information and Systems,2025,108(1):23-36.
[9] Gad G,Fadlullah Z M,Fouda M M,et al.Federated learning with selective knowledge distillation over bandwidth-constrained wireless networks[C]//International Conference on Communications,2024:3476-3481.
[10] Yang P,Yan M,Cui Y,et al.Communication-efficient federated double distillation in IoV[J].IEEE Transactions on Cognitive Communications and Networking,2023,9(5):1340-1352.
[11] CHEN J,ZHANG J.Personalized federated learning algorithm based on knowledge distillation without data[J].Information Network Security,2024,24(10):1562-1569.
[12] Zhang Y,Zhang W,Pu L,et al.To distill or not to distill:toward fast,accurate,and communication-efficient federated distillation learning[J].IEEE Internet of Things Journal,2023,11(6):10040-10053.
[13] LeCun Y,Bottou L,Bengio Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,2002,86(11):2278-2324.
[14] Cohen G,Afshar S,Tapson J,et al.EMNIST:extending MNIST to handwritten letters[C]//Interrnational Joint Conference on Neural Networks,2017:2921-2926.
[15] Caldas S,Duddu S M K,Wu P,et al.Leaf:a benchmark for federated settings[J].arXiv preprint arXiv:1812.01097,2018.
[16] Krizhevsky A,Hinton G.Learning multiple layers of features from tiny images[J].Handbook of Systemic Autoimmune Diseases,2009,1(4):1-60.
[17] Jiang Y,Wang S,Valls V,et al.Model pruning enables efficient federated learning on edge devices[J].IEEE Transactions on Neural Networks and Learning Systems,2022,34(12):10374-10386.
[18] Zhu Z,Shi Y,Luo J,et al.Fedlp:Layer-wise pruning mechanism for communication-computation efficient federated learning[C]//IEEE International Conference on Communications,2023:1250-1255.
[19] Huang H,Zhuang W,Chen C,et al.Fedmef:towards memory-efficient federated dynamic pruning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2024:27548-27557.

附中文参考文献:
[3] 王瑞锦,王金波,张凤荔,等.联邦原型学习的特征图中毒攻击和双重防御机制[J].软件学报,2024,36(3):1355-1374.
[5] 陈 晓,仇洪冰,李燕龙.边缘辅助的自适应稀疏联邦学习优化算法[J].电子与信息学报,2025,47(3):1-12.
[11] 陈 婧,张 健.基于知识蒸馏的无数据个性化联邦学习算法[J].信息网络安全,2024,24(10):1562-1569.

基金资助

国家自然科学基金项目 (62271128,U2333207)资助;四川省科技计划重点研发项目(2022ZDZX0004,2025YFHZ0302)资助;四川省科技计划“揭榜挂帅”项目(2023YFG0374)资助;成都重点研发支持计划项目(2025-YF08-00128-GX,2025-YF12-00029-RC)资助.