文图跨模态知识蒸馏的行人检索算法

Research on person re trieval algorithm based on text-to-image cross-modal knowledge distillation

摘要

当前文图跨模态行人检索算法过于依赖视觉语言预训练大模型来提升精度，导致模型参数量庞大、算力要求高等问题，难以满足边缘部署等实际应用需求。基于此，该文从视觉语言预训练大模型的轻量化角度入手，引入多模态联合蒸馏与互补式监督策略，基于华为昇腾平台，提出了一种基于三阶段渐进式知识蒸馏的文图跨模态行人检索算法。该方法区别于传统两阶段仅对各个模态进行独立蒸馏，而是通过模态内−跨模态的蒸馏路径，依次在图像、文本模态内实现特征对齐，最终在共享隐空间中进行跨模态语义关联的协同蒸馏。实验表明，学生模型参数量仅为教师模型的14.77%，在CUHK-PEDES、ICFG-PEDES和RSTPReid数据集上与现有轻量化方法相比，其mAP指标均得到有效提升，同时在边缘设备上处理单个文本−图像对的耗时约23 ms，实现了实时推理性能。该研究证实了基于文图跨模态知识蒸馏的行人检索算法的有效性，为人员排查、智能安防等场景国产化落地提供了有效路径。

Abstract

Current text-to-image cross-modal person retrieval algorithms overly rely on vision-language pre-trained large models to improve accuracy. This leads to issues such as large model parameter sizes and high computational requirements, making it difficult to meet the practical application requirements such as edge deployment. In light of this, this paper focuses on the lightweight aspect of vision-language pre-trained large models. It introduces a multi-modal joint distillation and complementary supervision strategy. Based on Huawei Ascend platform, the paper proposes a text-image cross-modal pedestrian retrieval algorithm based on a three-stage progressive knowledge distillation approach. Unlike the traditional two-stage approach that only performs independent distillation for each modality, this method first conducts distillation within the image and text modalities separately, and finally performs collaborative distillation for cross-modal semantic association in the shared latent space. Experimental results show that the student model has only 14.77% of the parameters of the teacher model. Compared with existing lightweight methods, it achieves effective improvements in mAP on the CUHK-PEDES, ICFG-PEDES, and RSTPReid datasets. It also realizes real-time inference performance with an inference time of approximately 23 ms for processing a single text-image pair on edge devices. This study confirms the effectiveness of the three-stage progressive knowledge distillation method and provides an effective pathway for the domestic implementation in scenarios such as personnel screening and intelligent security.

关键词

文图跨模态 / 行人检索 / 知识蒸馏 / 边缘设备

Key words

text-to-image cross-modal / person retrieval / knowledge distillation / edge devices

参考文献

[1] Li Shuang, Xiao Tong, Li Hongsheng, et al. Person search with natural language description[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5187-5196.

[2] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc., 2012: 1097-1105.

[3] He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.

[4] Hochreiter S, Schmidhuber J. Long short-term memory[J].Neural Computation, 1997, 9(8): 1735-1780.

[5] Zheng Zhedong, Zheng Liang, Garrett M, et al. Dual-path convolutional image-text embeddings with instance loss[J].ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2020, 16(2): 51.

[6] Ma Tinghuai, Yang Mingming, Rong Huan, et al. Dual-path CNN with max gated block for text-based person re-identification[J].Image and Vision Computing, 2021, 111: 104168.

[7] Farooq A, Awais M, Kittler J, et al. AXM-Net: Implicit cross-modal feature alignment for person re-identification[C]//Proceedings of the 36th AAAI Conference on Artificial Intelligence. [S.l.]: AAAI,2022: 4477-4485.

[8] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. [S.l.]: PMLR,2021: 8748-8763.

[9] Jia Chao, Yang Yinfei, Xia Ye, et al. Scaling up visual and vision-language representation learning with noisy text supervision[C]//Proceedings of the 38th International Conference on Machine Learning. [S.l.]: PMLR,2021: 4904-4916.

[10] Li Junnan, Selvaraju R R, Gotmare A D, et al. Align before fuse: Vision and language representation learning with momentum distillation[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. [S.l.]: Curran Associates Inc.,2021: 742.

[11] Li Junnan, Li Dongxu, Xiong Caiming, et al. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//Proceedings of the 39th International Conference on Machine Learning. Baltimore: PMLR, 2022: 12888-12900.

[12] Yang Shuyu, Zhou Yinan, Zheng Zhedong, et al. Towards unified text-based person retrieval: A large-scale multi-attribute and language search benchmark[C]//Proceedings of the 31st ACM International Conference on Multimedia. Ottawa: ACM, 2023: 4492-4501.

[13] Jiang Ding, Ye Mang. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval[C]//Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 2787-2797.

[14] Bai Yang, Cao Min, Gao Daming, et al. RaSa: Relation and sensitivity aware representation learning for text-based person search[C]//Proceedings of the 32nd International Joint Conference on Artificial Intelligence. Macao, China: IJCAI, 2023: 62.

[15] 石瑞鑫, 智敏, 殷雁君 . 多模态行人重识别研究综述[J].计算机应用研究, 2025, 42(7): 1921-1929.

[16] Shi Ruixin, Zhi Min, Yin Yanjun . Review of multimodal pedestrian re-identification[J].Application Research of Computers, 2025, 42(7): 1921-1929. (in Chinese)

[17] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[EB/OL]. (2015-03-09) [2025-10-11].https://arxiv.org/abs/1503.02531.

[18] Andonian A, Chen Shixing, Hamid R. Robust cross-modal representation learning with progressive self-distillation[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 16409-16420.

[19] Wu Kan, Peng Houwen, Zhou Zhenghong, et al. TinyCLIP: CLIP distillation via affinity mimicking and weight inheritance[C]//Proceedings of 2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 21913-21923.

[20] Ren Siyu, Zhu K. Leaner and faster: Two-stage model compression for lightweight text-image retrieval[C]//Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle: ACL, 2022: 4085-4090.

[21] Wang Jiapeng, Wang Chengyu, Wang Xiaodan, et al. CocaCLIP: Exploring distillation of fully-connected knowledge interaction graph for lightweight text-image retrieval[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Toronto: ACL, 2023: 71-80.

[22] Yang Chuanguang, An Zhulin, Huang Libo, et al. CLIP-KD: An empirical study of clip model distillation[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 15952-15962.

[23] Yang Kaicheng, Gu Tiancheng, An Xiang, et al. CLIP-CID: Efficient clip distillation via cluster-instance discrimination[C]//Proceedings of the 39th AAAI Conference on Artificial Intelligence. Philadelphia: AAAI, 2025: 21974-21982.

[24] 邵仁荣, 刘宇昂, 张伟, 等. 深度学习中知识蒸馏研究综述[J].计算机学报, 2022, 45(8): 1638-1673.

[25] Shao Renrong, Liu Yuang, Zhang Wei, et al. A survey of knowledge distillation in deep learning[J].Chinese Journal of Computers, 2022, 45(8): 1638-1673. (in Chinese)

[26] Chen Dapeng, Li Hongsheng, Liu Xihui, et al. Improving deep visual representation for person re-identification by global and local image-language association[C]//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018: 56-73.

[27] Zhang Ying, Lu Huchuan. Deep cross-modal projection learning for image-text matching[C]//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018: 707-723.

[28] Howard A G, Zhu Menglong, Chen Bo, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications[EB/OL]. (2017-04-17) [2025-10-11].https://arxiv.org/abs/1704.04861.

[29] Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures[J].Neural Networks, 2005, 18(5/6): 602-610.

[30] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 6000-6010.

[31] Romero A, Ballas N, Kahou S E, et al. FitNets: Hints for thin deep nets[C]//Proceedings of the 3rd International Conference on Learning Representations. San Diego: ICLR, 2015: 1-13.

[32] Yim J, Joo D, Bae J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE,2017: 7130-7138.

[33] Zhang Ying, Xiang Tao, Hospedales T M, et al. Deep mutual learning[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4320-4328.

[34] Zhang Linfeng, Song Jiebo, Gao Anni, et al. Be your own teacher: Improve the performance of convolutional neural networks via self distillation[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3712-3721.

[35] Yang Chenglin, Xie Lingxi, Su Chi, et al. Snapshot distillation: Teacher-student optimization in one generation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2854-2863.

[36] van den Oord A, Li Yazhe, Vinyals O. Representation learning with contrastive predictive coding[EB/OL]. (2018-07-10) [2025-10-11].https://arxiv.org/abs/1807.03748.

[37] Ding Zefeng, Ding Changxing, Shao Zhiyin, et al. Semantically self-aligned network for text-to-image part-aware person re-identification[EB/OL]. (2021-07-27) [2025-10-11].https://arxiv.org/abs/2107.12666.

[38] Zhu Aichun, Wang Zijie, Li Yifeng, et al. DSSL: Deep surroundings-person separation learning for text-based person retrieval[C]//Proceedings of the 29th ACM International Conference on Multimedia. [S.l.]: ACM,2021: 209-217.

[39] Kingma D P, Ba J. Adam: A method for stochastic optimization[C]//Proceedings of the 3rd International Conference on Learning Representations. San Diego: ICLR, 2015.

[40] Wang Feng, Liu Huaping. Understanding the behaviour of contrastive loss[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 2495-2504.

[41] Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 618-626.

基金资助

国家自然科学基金(62372082)

中央高校基本科研业务费(ZYGX2024Z017)

深圳市自然科学基金(JCYJ20240813114206010)